Yahoo Answers is shutting down on May 4th, 2021 (Eastern Time) and beginning April 20th, 2021 (Eastern Time) the Yahoo Answers website will be in read-only mode. There will be no changes to other Yahoo properties or services, or your Yahoo account. You can find more information about the Yahoo Answers shutdown and how to download your data on this help page.
Trending News
What is the porpose of Robot.txt file in SEO?
2 Answers
- 1 year ago
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
The main purpose of Robot.txt file is that it tells which pages on your site to crawl.
or It also tells web robots which pages not to crawl.
- Anonymous1 year ago
simply put, it's used to direct search engines to the appropriate content on your site
