Robots.txt is a file that gives instructions to the spiderbots of search engines like Google, Yahoo and Bing on how they may index different websites. - It is to be placed in the root directory. - Spiders then adhere to the instructions when they explore your website, so you can effectively control how they crawl and index your webpages.
Home
Search
Login
Register