Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-26 09:04:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

added robots.txt question

Browse source 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@932 e7d03fd6-7b0d-0410-9947-9c21f3af8025
This commit is contained in:
calvin 2003-07-04 14:24:36 +00:00
parent b3ec5ddb2f
commit 0e4fca4572
1 changed files with 6 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
FAQ
View file

@ -138,3 +138,9 @@ A10: This is not a bug.
This is the reason for always checking URLs with different anchors.
If you really want to disable this, use --no-anchor-caching.
Q11: I see linkchecker gets a "/robots.txt" file for every site it
checks. What is that about?
A11: See http://www.robotstxt.org/wc/robots.html and
http://www.w3.org/Search/9605-Indexing-Workshop/ReportOutcomes/Spidering.txt
for more info.