|
|
Robots.txt
A robots.txt file is a document used by webmasters to control the crawling of a website by search engines. Search engines which follow the protocol may be instructed not to crawl either all or part of a websites as specified in the document.
Crawl instructions can be given generally to all robots / spiders crawling the website, or specifically to individual bots. For example, if a webmaster did not want a particular search engine or other service to crawl the site, this could be specified in the robots.txt. This may be used to prevent website data being used as competitive information, such as with the Linkscape service from SEOmoz.
Despite the wide acceptance of the protocol, there is no guarantee that search engine robots will comply with webmaster requests not to crawl areas of a website. As such, no data that is not intended to be indexed should be placed on a public server where it may be indexed.
Robots.txt files are simple, plain text documents and can be created with a basic text editor such as Notepad. For a website wishing to keep robots from crawling an entire site, a form like this might be used:
User-agent: *
Disallow: /
To prevent a robot from crawling certain directories, the file would look something like this (where "/cgi-bin/" and "/images/" were the directories to be blocked):
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
To use a robots.txt file, ensure it is placed in a website's root directory, where it can be found by spiders accessing the site, for example, http://www.xyz.com/robots.txt.
|
peace of mind
" iView Multimedia, a wholly owned subsidiary of Microsoft, came to Backbone IT Group to improve their website rankings for their most difficult search terms- to find out what happened click here"  
total assurance
Your SEO results are backed by our money back guarantee.
If any of your guaranteed search terms do not place in the Top 10 results on at least one of the three major search engines (either MSN, Yahoo or Google) we will refund your monthly payment for that search term in full.
Find out how you could benefit from a high quality search engine optimization service tailored exactly to your needs. Talk to us now. No pressure, no obligation. Free SEO quote.
Enquire Now
UK Direct Tel:
01524 65533
UK 24hr Local Tel:
0845 057 3371
US Toll Free:
1 877 413 1158
Email Us
|
| Email Us
