Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-20 22:31:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

added robots.txt example file

Browse source 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2193 e7d03fd6-7b0d-0410-9947-9c21f3af8025
This commit is contained in:
calvin 2005-01-24 09:38:32 +00:00
parent 7ab9ffd815
commit 6c83ebbcad
2 changed files with 12 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
debian/linkchecker.examples vendored
View file

@ -1 +1,2 @@
config/linkcheck-cron.sh
doc/robots.txt

11
doc/robots.txt Normal file
View file

@ -0,0 +1,11 @@
# Simple robots.txt example, put this into your web root.
# See the complete reference at
# http://www.robotstxt.org/wc/norobots-rfc.html
# disallow cgi-bin access for all robots
User-agent: *
Disallow: /cgi-bin/
# All LinkChecker versions are not allowed to check anything
User-agent: LinkChecker
Disallow: /