Retrevo’s Web Crawler

While working at Persistent Systems Limited, I had chance to work on a web crawler. Web mining is my all time favorite area. So I quickly jumped onto this opportunity. Working on this web crawler was a really challenging experience.

The technical scene was like this: The application was a desktop app, developed in Tcl/Tk. OpTcl extension was used to access Windows libraries. Basically being a visual crawler, this application needed to use Internet Explorer object. It was using DOM Events generated per page visited.