Erlang Bot (Ebot) is an opensource web crawler written on top of Erlang, a NOSQL database (Apache CouchDB or Riak), RabbitMQ, Webmachine (Mochiweb), RRDTOOL, .. Using a NOSQL instead of a Relational Database, Ebot can grow easily and cheaply… Ebot is a solid and highly scalable, distribuited and customizable web crawler.
Below there is an example of a url document generated by the ebot crawler (with apache couchdb backend)
Below you find a sample image of Statistics generated by ebot web crawler using RRDTOOL
you are welcome