Webbots? What are they??

Internet bots, also known as web robots, WWW robots or simply bots, are nothing but the software applications that run automated tasks over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone.

In general, web bots can do almost everything what a human can do over the Internet. They can simply:

• browse the web
• read the web pages
• fill up and submit the forms
• read the emails and respond automatically
• notice the changes in contents
• update their knowledge about something (like prices etc.)
• download something that satisfies particular criteria
• … and many other actions!

The largest use of web bots is in web spidering, in which an automated script fetches, analyzes and files information from web servers at many times the speed of a human. These programs are also called as web crawlers.

Where can I use them?

The web has become a way of life and is carrying virtually all important data of this world now-a-day. Utilities that bring specific target data to your computer are a vital tool for business of these days. Web bots can address this very need. The web scraping, web aggregation, web crawling and web automation activities are important and useful in many cases – and all of these can be carried out with the help of bots (with minimal human supervision).

Web Scraping
Sometimes we need to use some data available publicly on some web site. For example, prices of some products. But copy-pasting it each time to update our copy is not at all a good idea. We can make use of web scraping bots to do this task for us. Just seat back relaxed, and this scrapper will do everything on behalf of you.

Web Aggregation
Whenever it comes to decision making, we must compare something with the other. So, how do we do this – by visiting all the web pages which are at different locations and noting down the values? Why not to aggregate these?! Web aggregation bots can fetch the contents you bother about, from different web pages and give you a consolidated view of them. The contents from different sources will look exactly same as seen on the original site! Moreover, fetching the contents will be a live, runtime activity. So whenever source gets updated, you are also updated!

Web crawlers
Want a repository of particular type of resources over the web? Use a web crawler! Web crawlers will go and collect whatever you want from the web. The criteria can be anything: get me all mp3 files, or make a list of all pages which have phrase “social media marketing” in the contents, … literally anything! These bots are generally aided by regular expressions, and hence work very effectively. They can give you the list of locations where resources you want are, and they can even download them for you!

Web Watcher
One cannot avoid competition when it comes to business. To stay a step ahead of your competitors, one must keep a watch on them. Every business proudly publishes its Press Releases, Media Coverage, Product Introductions, White Papers, Customer Wins, Upcoming Events, Promotions, Price Changes, Job Openings etc. This information can tell you a lot about what going on at your competitor’s end. But do we need to invest any time to watch our competitors? No! Web bots are also applicable for this. These bots will eventually go and check whether anything has changed on your competitors’ web page. If yes, it will immediately let you know by an email alert!

Automation based on Emails

This is really interesting wing of web bots. You can set email-automation bots to read the emails on your behalf. They can forward the emails to others based on the subject line or sender account or email contents. Moreover, these bots can even reply back to the sender with some predefined message, if they find particular word/phrase/information in mail body or subject line!

Link Verifiers

These bots are basically meant for verifying whether all the links on the web page point to valid, existing and reachable web resource. These bots generally start with the home page, and go on following each link present on each web page. (avoiding the duplications, of course!) Whenever they find the resource being pointed is missing or out of reach (for example, web page cannot be seen, or video file not readable) they notify the user in some or the other way.

These are some of the major uses of web bots, but the list is not limited to! You name the activity over the Internet and most of the times it is doable with web bots. Yes, most of the times… There are still some cases where web bots fail to perform some action, especially when working with dynamically generated links – the one’s which have pseudo session id in the URL. The session ID never remains the same and bots fail in this case.

There are many ethics and other precautions needs to be followed by each web bot (at least expected to do so). Good web bots developers always follow these ethics – most important being to respect robots.txt file on each web server. If this file denies you from crawling/visiting any directory, then you should not! (I would write on this topic in a separate post.)

But overall, web bots are becoming really important factors in the world of the web!

In a nutshell, the protocol is something like this:
1. Feed URL declares its hub server(s) in its Atom/RSS xml file. It mentions <link rel=”hub” …>. The hub(s) can be run by the publisher of the feed itself, or it can also be a community hub that anybody can use.
2. A subscriber (i.e. the server who is interested in the feed) initially fetches the Atom URL as usual. If the Atom file declare its hub(s), the subscriber then can avoid the polling repeatitions. Instead, it can resiter with the feed’s hub(s) to get notified about the updates.
3. The subscriber now subscribes for ther target feed URL from the hub.
4. Whenever publisher updates the feed, it pings the hub(s) telling them about the update.
5. The hub now fetches the published updates and multicasts the new/changed update to all the registered subscribers for this feed!

You can see following video and/or slide show to get fair idea about PSHB protocol.

This project is hosted on Google code. Source code, wiki and some other doucments are also available there.

If you fire telnet or ftp command on 127.0.0.1, you will get connected to very same machine from which the command was fired. Suppose your machine’s name is ‘alchemist’. If you telnet to 127.0.0.1, you see something like this:

telnet 127.0.0.1
Trying 127.0.0.1…
Connected to alchemist
Escape character is ‘^]’

127.0.0.1 or localhost is used in place of the name of the computer to be connected.  Though, 127.0.0.1 is most commonly used address for localhost, any IP address in the range 127.*.*.* should work in the same way.

Making a connection to 127.0.0.1 is the same as connecting with any other address. The only difference is that, in this case the local network interface hardware is avoided. This address is widely used by developers with intentions to test some web-based application. For IPv4 connections, the loopback address of the machine is usually assigned the address 127.0.0.1, with subnet mask 255.0.0.0.

Proper use of the address 127.0.0.1 is defined in RFC 3330 – Special Use IPv4 Addresses. It says:

127.0.0.0/8 - This block is assigned for use as the Internet host
   loopback address.  A datagram sent by a higher level protocol to an
   address anywhere within this block should loop back inside the host.
   This is ordinarily implemented using only 127.0.0.1/32 for loopback,
   but no addresses within this block should ever appear on any network
   anywhere [RFC1700, page 5].

The IPv6 version of localhost is defined in RFC 3513 – Internet Protocol Version 6 Addressing Architecture. It is defined as ::1/128.

Listening is really underrated, most of the times. Try to hear what the client is saying. Ask them for details, thoughts on design, copy, layout, and then ask for more! Listen to what they say. Sometimes, listening between the lines is necessary too!

Then, after you listen completely, communicate to the client. Keep them updated through the process. Reply to emails as quickly as possible. Send them progress updates. Then, after you launch their site/write their piece, contact them a week or so later to touch the base and see if they need anything else.

Clients love this.

Simply download the Ophcrack ISO image and burn it into a CD. You can also load it onto an USB drive using UNetbootin. Insert this CD into machine to which you want to gain access to, then press and hold the power button down until the machine shuts down. Turn ON the computer again, and enter BIOS. Change the boot sequence to CD before HDD, save the change and exit.

The computer will restart again, and Ophcrack will get loaded. Now just relax and sit back! This software will do whatever it is required to do. Write down the password it tell you. That’s it! Remove the CD and reboot again. Use the given password to login.

Provide high quality contents on your website – especially the home page. If your page(s) really contain useful information, the readers will like to link to you. For creating helpful, information-rich website, describe each of your topics neatly and accurately. Think about the words people would type to search for your site.

2. Make sure other sites link to yours.

Links really help Google crawler to find your site. They provide a visibility to your site. Google have its own, special algorithm to “value” your site. A link from page A to page B is considered as a “vote by A to B”. If page A is really a higher valued page, you ultimately get a good value from Google’s point of view.

3. Google algorithms can distinguish between natural and unnatural links.

Natural links to your site develop a dynamic nature of web when other sites find your contents useful/valuable and think it would be helpful for their readers (For example, links provided in blog posts). Unnatural links are just put there specifically to “make” your site look popular. Google can differentiate between these two, and only the natural links are considered.

4. Make your site easily accessible.

Build your site with some good logical links structure. Every page should be reachable from at least one static text link. Use a text browser, such as Lynx, to examine your site. Most crawlers will “see” your site as Lynx do. If features such as JavaScript, cookies, session IDs, frames, DHTML, Flash etc keep you from seeing your entire site in text browser, then it will also keep Google and other crawlers from seeing your site.

5. Things you should avoid.

Do not fill your pages with some list of “keywords” in any attempt to put “crawler only” pages. If your site contains pages, links or text contents you wish your visitors should not see, Google considers such links and pages deceptive and it may ignore your site.

Avail some search engine optimization service. But be careful while choosing one! Many such services claim to “guarantee” top position in Google results, and they do so by means of either improving your contents and flow, or by employing some deceptive strategy attempting to fool Google. But if your domain is affiliated with such any service, Google may ban you from their index.

Do not use images to display important names, contents or links. Google’s crawler cannot recognize text contained in graphics. Use ALT attribute if main names or contents on your pages cannot be formatted in simple HTML.

You can use this utility to monitor your competitor’s web pages (like products page, press releases and news, customer wins etc). Whenever this utility finds any change on web pages being monitored, it alerts you, as well as it finds out all the changed locations and shows you the actual change on pages by highlighting them! Thus, you need not worry about what exactly has changed – as this utility is capable of showing it to you.

This utility is being developed as a desktop application. Hence, you need not keep the browsers open all the time, nor you need to manually refresh them or employ anybody to do this checking for you. Install this, and that’s it!

I have designed this utility in such a way that it consumes very lesser space on your desktop. It also have many configurable options such as, time interval between two subsequent checks, when to notify you (ranging from ‘Notify Always’ to ‘Notify only if change found’ to ‘Never Notify’). Follow Bee is planned to get released on 26th January 2010, but there can be a slight change in the release date, if needed.

It collects many statistics regarding site visits, including IP addresses, page hits, site hits per IP address, hour-wise hits etc. At the end of the day, this service sends a consolidated email report to your email address. You can then make use of this report to decide how appealing your site is to users.

As the customer was also from the technical field, I could develop GUI which accepts the starting point – start URL – and the regular expression to check against. Once the application starts crawling the given website, it checks each hyper link and if it is matching with the given expression, it adds it to the bucket. In the end, it exports the result bucket as CVS file.

Some web sites detect the automated program by observing the time duration between two requests sent for different web pages on the site. And if they suspect any IP address sending such requests, they simply deny the contents. This application is designed so as to work in such situations as well. The application waits for random number of seconds (3 to 7 seconds) before sending next request. Thus, it imitates human browsing.

I started using it actively and gradually developed an avid interest. I think Twitter is a manageable process that can be adopted for all types of lifestyles, busy or inactive. You’ll just need to integrate it within your normal workflow. It’s addictive but once you understand how to use it, it can be a very effective tool indeed!

Having read a great deal of other articles on Twitter, I decided a do a quick summary of all the ideas you can use Twitter for – your professional and/or personal life. Some of these methods go beyond the use of Twitter as a lifestreaming device:

1. Use it as a Personal Branding Tool. Twitter is a social media platform which you can use to build your personal brand. It establishes you as a social personality that is connected and approachable. As Twitter adoption increases, new users will be drawn towards well established Twitter personas.
2. Get Feedbacks! If you need another opinion on how a website looks or the right course of action to take, just drop out a message asking for advice and you’ll receive replies from other users. This collective intelligence can be used as feeds  for articles or projects.
3. Attract traffic. Twitter can be used to get traffic to your websites or the sites of your friends. If you ask your friends to tweet about it, the message will spread faster. There is a viral nature to all types of news, even on a site like Twitter.
4. Tweeting Newspaper. Twitter users often link to useful sites or articles and can be a source of scoops and alternative news. You can also subscribe to Twitter feeds for specific websites or conferences, which allows you to receive and view content quickly. This is very useful for active social news participants.
5. Making New Friends. Like any other social networking site, Twitter has a built-in function for you to befriend and track the messages of other users. This is an easy way for you connect with people outside of your usual circle. Take some efforts to add active users you find interesting. A Twitter acquaintance can be developed into a long lasting friendship!
6. Network for benefits. Twitter can be used as a socializing platform for you to interact with other like-minded people, especially those in the same industry. It can be used to establish consistent and deeper relationships for future benefits such as testimonials or peer recommendations.
7. Business Management. Twitter can be used as a company intranet that connects employees to one another. Employees can interact with each other when working on group projects. Particularly useful when certain workers go out often in the field. Updates could be set to private for security reasons.
8. A Mean To Notify Your Customers. Set up a Twitter feed for the specific purpose of notifying customers when new products come in. Customers can subscribe via mobile or RSS for instant notification. Twitter can also be used to provide mini-updates for one-on-one clients.
9. Take Notes. Twitter provides you with an easy way to record important ideas or concepts you want to explore further. Include links relevant to ideas you want to explore. Note taking can also be done offline via mobile applications.
10. Event Updates. Businesses can use Twitter as a means to inform event participants and latest event happenings/changes. This is a hassle-free way of disseminating information, especially when you don’t have the means to set up a direct mobile link between you and the audience. As such its applicable to almost all event organizers.
11. Find Prospects. Twitter can be used as a means to find potential customers or clients online. Do a search for keywords related to your product on Twitter Search and then follow users. Tweet about topics parallel to your product and close prospects away from public channels by using direct messages or offline communications. Discretion and skill is needed in this area.
12. Provide Live Coverage. Twitter’s message size limit prevents detailed coverage of events but it can allow you to provide real-time commentary which may help to spark further discussion or interest on the event as other Twitter users spread the message. Very useful for citizen journalism.
13. Time Management and Analysis. Twitter can simply be used to keep a detailed record of what you are doing every daily. This might be boring for others but this type of usage is useful when you want to analyze how you spend and manage your time.
14. Set Up Meetings. Twitter can help you organize impromptu meetups. For example, you can twitter a message while at a cafe, event or art gallery and arrange to meet fellow users at a specific spot. It’s an informal and casual way of arranging a meeting.
15. Acquire Votes. Send a link to your stories you’ve submitted in other social news sites like Digg. Sometimes your followers will vote up the stories because they agree with it. This allows you to acquire more support for your efforts on other social media websites.

I suspect I’ll be writing a little more about Twitter in the future including a roundup and summary of some of the articles I found really useful (there were quite a few).

Hopefully this article gave you ideas on how to use Twitter. If you found this article useful, feel free to share it with your Twitter friends.