Follow Bee is a web contents monitoring utility, meant to aid competitive intelligence (CI). This utility checks the web pages after given time interval, and reports you if it finds any change in the web page contents. The intended users of Follow Bee are higher level executives, marketing personnel, business development executives and all others, whose main information source is the web.

You can use this utility to monitor your competitor’s web pages (like products page, press releases and news, customer wins etc). Whenever this utility finds any change on web pages being monitored, it alerts you, as well as it finds out all the changed locations and shows you the actual change on pages by highlighting them! Thus, you need not worry about what exactly has changed – as this utility is capable of showing it to you.

This utility is being developed as a desktop application. Hence, you need not keep the browsers open all the time, nor you need to manually refresh them or employ anybody to do this checking for you. Install this, and that’s it!

I have designed this utility in such a way that it consumes very lesser space on your desktop. It also have many configurable options such as, time interval between two subsequent checks, when to notify you (ranging from ‘Notify Always’ to ‘Notify only if change found’ to ‘Never Notify’). Follow Bee is planned to get released on 26th January 2010, but there can be a slight change in the release date, if needed.

This is a web-based service, developed to track daily visits to your website. You just need to include a small javascript inside your website’s home page, and the script takes care of everything else.

It collects many statistics regarding site visits, including IP addresses, page hits, site hits per IP address, hour-wise hits etc. At the end of the day, this service sends a consolidated email report to your email address. You can then make use of this report to decide how appealing your site is to users.

One of my customers wanted something to crawl complete website and collect all the links matching with given regular expression. So I came up with this website crawler application.

As the customer was also from the technical field, I could develop GUI which accepts the starting point – start URL – and the regular expression to check against. Once the application starts crawling the given website, it checks each hyper link and if it is matching with the given expression, it adds it to the bucket. In the end, it exports the result bucket as CVS file.

Some web sites detect the automated program by observing the time duration between two requests sent for different web pages on the site. And if they suspect any IP address sending such requests, they simply deny the contents. This application is designed so as to work in such situations as well. The application waits for random number of seconds (3 to 7 seconds) before sending next request. Thus, it imitates human browsing.

This is a small windows application, to scrap result set returned by Google. This small app accepts a search query and a regular expression, and then collects all links from the result set which match with given regular expression.

The customer wanted to check how many of his articles submitted across many forums are getting indexed on Google. He wanted the links to those forum posts as on Google’s records in the output. The customer being from the technical field, he could write regular expression to implement his criteria (for example, all pages having .html as extension).

Google is capable of detecting whether the HTTP requests are coming as a result of human browsing or bot program. Hence this app was programmed to wait for 3 seconds before clicking ‘Next’ link. Thus, it succeeds in scrapping all the 100 result pages returned by Google.

ype: Application
Tech: C#.Net, MS SQL Server, XML
Role: Design, Development and Testing

Type: Web aggregation
Tech: PHP-MySQL on IIS Server
Role: Design, Development and Testing

Type: Web scraping
Tech: PHP-MySQL on IIS Server
Role: Development, Maintenance and Testing

Type: Application
Tech: Ruby-MySQL on Linux based Apache Server
Role: Design, Development and Testing

Type: Web-based Wizard, operated in Intranet.
Tech: Ruby on Rails-MySQL on Linux based Apache Server
Role: Enhancements, Maintenance and Testing

While working at Persistent Systems Limited, I had chance to work on a web crawler. Web mining is my all time favorite area. So I quickly jumped onto this opportunity. Working on this web crawler was a really challenging experience.

The technical scene was like this: The application was a desktop app, developed in Tcl/Tk. OpTcl extension was used to access Windows libraries. Basically being a visual crawler, this application needed to use Internet Explorer object. It was using DOM Events generated per page visited.