Follow Bee is a web contents monitoring utility, meant to aid competitive intelligence (CI). This utility checks the web pages after given time interval, and reports you if it finds any change in the web page contents. The intended users of Follow Bee are higher level executives, marketing personnel, business development executives and all others, whose main information source is the web.

You can use this utility to monitor your competitor’s web pages (like products page, press releases and news, customer wins etc). Whenever this utility finds any change on web pages being monitored, it alerts you, as well as it finds out all the changed locations and shows you the actual change on pages by highlighting them! Thus, you need not worry about what exactly has changed – as this utility is capable of showing it to you.

This utility is being developed as a desktop application. Hence, you need not keep the browsers open all the time, nor you need to manually refresh them or employ anybody to do this checking for you. Install this, and that’s it!

I have designed this utility in such a way that it consumes very lesser space on your desktop. It also have many configurable options such as, time interval between two subsequent checks, when to notify you (ranging from ‘Notify Always’ to ‘Notify only if change found’ to ‘Never Notify’). Follow Bee is planned to get released on 26th January 2010, but there can be a slight change in the release date, if needed.

One of my customers wanted something to crawl complete website and collect all the links matching with given regular expression. So I came up with this website crawler application.

As the customer was also from the technical field, I could develop GUI which accepts the starting point – start URL – and the regular expression to check against. Once the application starts crawling the given website, it checks each hyper link and if it is matching with the given expression, it adds it to the bucket. In the end, it exports the result bucket as CVS file.

Some web sites detect the automated program by observing the time duration between two requests sent for different web pages on the site. And if they suspect any IP address sending such requests, they simply deny the contents. This application is designed so as to work in such situations as well. The application waits for random number of seconds (3 to 7 seconds) before sending next request. Thus, it imitates human browsing.

This is a small windows application, to scrap result set returned by Google. This small app accepts a search query and a regular expression, and then collects all links from the result set which match with given regular expression.

The customer wanted to check how many of his articles submitted across many forums are getting indexed on Google. He wanted the links to those forum posts as on Google’s records in the output. The customer being from the technical field, he could write regular expression to implement his criteria (for example, all pages having .html as extension).

Google is capable of detecting whether the HTTP requests are coming as a result of human browsing or bot program. Hence this app was programmed to wait for 3 seconds before clicking ‘Next’ link. Thus, it succeeds in scrapping all the 100 result pages returned by Google.

ype: Application
Tech: C#.Net, MS SQL Server, XML
Role: Design, Development and Testing

Type: Application
Tech: Ruby-MySQL on Linux based Apache Server
Role: Design, Development and Testing

While working at Persistent Systems Limited, I had chance to work on a web crawler. Web mining is my all time favorite area. So I quickly jumped onto this opportunity. Working on this web crawler was a really challenging experience.

The technical scene was like this: The application was a desktop app, developed in Tcl/Tk. OpTcl extension was used to access Windows libraries. Basically being a visual crawler, this application needed to use Internet Explorer object. It was using DOM Events generated per page visited.

This is a backup software and system project. The project started in July 2006 with researches and ended up in March 2007, with version 1 ready.

The major goals for this project were:

• come up with a backup software which will back up the files efficiently
• support various kind of files, like office documents, large database files, media files etc.
• support for transfer over telephone lines, along with via Internet
• should be easy to use, handy to non-technical staff

The project consists of two major parts – the software with GUI and the iSCSI initiators and targets. The software, along with iSCSI initiators needed to be installed on each client machine in the network, whereas the iSCSI target would sit at backup site – the machine where all the data from clients will get consolidated. The Rsync protocol was used to update the file copies appropriately.

The iSCSI targets were deployed on footprint versions of Linux operating system. In order to make them easy to manage, the Webmin interface was customized.

The telephony support was added with TAPI programming.

While working on the GUI, I used the theme-scheme similar to that of Windows, so that it will “look” familiar to the user. The menus were exactly same as those of normal MS Office applications. The source selection, backup scheduling and immediate backup wizards were developed using KISS principle, making them as simple as possible to use. Source selection and backup scheduling were kept separated (instead of keeping in a single flow), so as to avoid a lengthy set of actions. The user could select the source files as and when he want, and he could also schedule a backup for these files any point in time later.

Webmin being the web-based administration for Linux systems, was selected to manage the iSCSI targets and LUNs (Logical Unit Numbers). Hence, one could configure the consolidation target machine through the browser itself, from any machine in the network.