Hound: Website crawler

Hound is a website crawler i developed a couple of months ago. Today i'm releasing the 0.11 version, which includes some bug fixes and new features.

The crawler starts by crawling a given base URL. It then analyses its html code and searches for other URLs which will be collected and will be enqueued for analysis.

The crawler's behaviour is based on plugins. Different kinds of plugins affect it in a different way. One can, for example, activate certain Filter Plugins which will restrict the URLs the crawler will visit, based on each plugin's behaviour. It could be undesirable, under most circumstances, to allow Hound to visit google, facebook, or youtube. This is why a HostFilter can be used, making the crawler only visit URLs that belong to the base host.

There are different types of plugins, each executed on a certain phase of the crawling session. These can be:

• Parsers: These are applied to downloaded html in order to normalise it so that the other plugins are able to collect data without worrying about aspects such as which encoding is used, html entities.
• Crawl filters: These are applied to every URL found. If a crawl filter matches a certain URL, then the latter is discarded. Host, extension and network filters are examples of them.
• Collect filters: These filters are applied to collected URLs, so that they are not taken into account in the crawling results. Again, you might not want to include google links in the results.
• Form collect filters: These filters are applied to form tags found in html files.
• Header filters: These are applied to a downloaded file's headers. They are usually used when filtering mime-types, for example.
• Collectors: The most important plugins. These take care of analysing the html files, parsing href or src attributes, among others, and feeding the crawler with new URLs.

The file hound.conf contains a list of active plugins, including their arguments. Once you've picked the right configuration, you can start a crawling session by executing:

./hound http://website.to.crawl.com

 The output will be only be written to stdout. If you want to store it in a file, you can do it by using the -o parameter, followed by its path. This will write the results both to stdout and to the given file. If you don't want to write results to stdout, use the -n parameter.

Once the crawl session has ended, you can use hound to parse the results. Run the following command to list the URLs found:

./hound -i /tmp/hound.out -p urls

Where /tmp/hound.out is the output file used during the crawling session. You can always parse the results manually, since they're stored in text files. To list all the form tags found, execute:

./hound -i /tmp/hound.out -p forms

Which will print something like:

0 POST http://blablabla/search cms_search --- hidden +++ query --- text +++ commit --- image
1 POST http://blablabla/contact/send article_id --- hidden +++ subject --- text +++ sender_name --- text +++ sender_mail --- text +++ reset --- reset

The number on the left of each line identifies each form. This is the way hound uses to encode forms. To generate the html code for a given form id, run:

./hound -i /tmp/hound.out -p form:0

Which will print the html code for the first form.

In order to download hound, you can visit the sourceforge project's site. It is open source, developed using python so you can have a look at the code and create new plugins to serve your needs.

Simple socks5 server in C++

This is a socks5 proxy server i implemented a few months ago. It's developed in C++, and as far as i have tested, works pretty well. There are a couple of things left to do, like handling domain name connection requests, or do a better thread handling, but it has served its purpose so far.

In order to compile it, you can use the GNU C++ compiler, and linking the application with libpthread:

g++ -o socks5 socks5.cpp -lpthread

By default it only accepts authenticated connections, using the USERNAME define as username, and the PASSWORD define as password. If you want to allow unauthenticated connection requests, add a -DALLOW_NO_AUTH flag when compiling.

The proxy server listens on port 5555, you might as well want to change the SERVER_PORT define if you want it to wait for connections on another port.

The source code can be downloaded from github: https://github.com/mfontanini/Programs-Scripts/blob/master/socks5/socks5.cpp.

Hope you find it useful!