Google Results Scrapper
This is a small windows application, to scrap result set returned by Google. This small app accepts a search query and a regular expression, and then collects all links from the result set which match with given regular expression.
The customer wanted to check how many of his articles submitted across many forums are getting indexed on Google. He wanted the links to those forum posts as on Google’s records in the output. The customer being from the technical field, he could write regular expression to implement his criteria (for example, all pages having .html as extension).
Google is capable of detecting whether the HTTP requests are coming as a result of human browsing or bot program. Hence this app was programmed to wait for 3 seconds before clicking ‘Next’ link. Thus, it succeeds in scrapping all the 100 result pages returned by Google.
Application | Prabhas Gupte | 
May 7, 2009 4:21 pm | 
Comments (0)
Subscribe