diff --git a/doc/en/documentation.html b/doc/en/documentation.html index c6e09305..8e3ef274 100644 --- a/doc/en/documentation.html +++ b/doc/en/documentation.html @@ -277,7 +277,7 @@ If you really want to disable this, use the

Q: I see LinkChecker gets a /robots.txt file for every site it checks. What is that about?

-

A: LinkChecker follows the robots.txt exclusion standard. To avoid +

A: LinkChecker follows the robots.txt exclusion standard. To avoid misuse of LinkChecker, you cannot turn this feature off. See the Web Robot pages and the Spidering report for more info.

Q: Ctrl-C does not stop LinkChecker immediately. Why is that so?

@@ -297,7 +297,7 @@ and look for missing files.

diff --git a/doc/en/documentation.txt b/doc/en/documentation.txt index 0ccd53f9..cb52cdf8 100644 --- a/doc/en/documentation.txt +++ b/doc/en/documentation.txt @@ -299,10 +299,12 @@ option. **Q: I see LinkChecker gets a /robots.txt file for every site it checks. What is that about?** -A: LinkChecker follows the robots.txt exclusion standard. To avoid +A: LinkChecker follows the `robots.txt exclusion standard`_. To avoid misuse of LinkChecker, you cannot turn this feature off. See the `Web Robot pages`_ and the `Spidering report`_ for more info. +.. _`robots.txt exclusion standard`: + http://www.robotstxt.org/wc/norobots-rfc.html .. _`Web Robot pages`: http://www.robotstxt.org/wc/robots.html .. _`Spidering report`: