spiders in the web site

[sarj]

After looking who was online, I was just curious what an AskJeeves Spider and an MSNBot Spider were.

 

[Steve St.Laurent]

Those are computers from those search engines that crawl sites so that stuff shows up in their search engines.

 

[CUMMINZ]

A little better explaination.



Every website literally linked onto the backbone of the internet worldwide uses a URL (ip address). Also each page on that site whether seen or unseen is accessible to every other computer in the world. Major search engines have computer (software) that it is there job to call on every IP address to record every page and change to that page in it's "index". These spiders record every word on each page and especially the "meta tags" which you can not see online. These meta tags contain site name, content & key words and a whole lot more. The programmer of the website can set perameters in their site that will exclude hidden files and pages and other info they do not want spidered. A popular site like TDR could very easily be spidered many times a week and often daily. The purpose for a personal firewall is to keep out these unwanted pings. A firewall on a server hosting websites can set their firewall to except spiders.