Yahoo Answers is shutting down on May 4th, 2021 (Eastern Time) and beginning April 20th, 2021 (Eastern Time) the Yahoo Answers website will be in read-only mode. There will be no changes to other Yahoo properties or services, or your Yahoo account. You can find more information about the Yahoo Answers shutdown and how to download your data on this help page.
Trending News
Promoted
Is there a php script that takes a URL, and then tells the script whether the URL should be read or not?
This is with regard to complying with the robots exclusion standard (robots.txt) that a site may have. I'm trying to write a compliant bot, FWIW.
2 Answers
Relevance
- Robin TLv 51 decade agoFavorite Answer
Just search for 'php robot parser' with your favorite search engine.
- Anonymous1 decade ago
There will be - as soon as you write it. If you can't, you're not up to writing any kind of bot.
Still have questions? Get your answers by asking now.