Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-05 05:04:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Document user-agent string.

Browse source 
[ci skip]
This commit is contained in:
Bastian Kleineidam 2014-09-05 22:47:26 +02:00
parent 7239cd1b76
commit c580c0ec52
1 changed files with 5 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
doc/web/content/faq.md
View file

@ -19,6 +19,11 @@ policy by the webmaster running the website you are checking. Look in
the ``/robots.txt`` file which follows the
[robots.txt exclusion standard](http://www.robotstxt.org/robotstxt.html).
For identification LinkChecker adds to each request a User-Agent header
like this:
Mozilla/5.0 (compatible; LinkChecker/9.3; +http://wummel.github.io/linkchecker/)
If you yourself are the webmaster, consider allowing LinkChecker to
check your web pages by adding the following to your robots.txt file: