Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-03-16 22:10:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add non-ascii values to test robots.txt

Browse source 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3851 e7d03fd6-7b0d-0410-9947-9c21f3af8025
This commit is contained in:
calvin 2008-07-13 13:01:59 +00:00
parent 7b2a21c133
commit bb8bca1b70
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
robots.txt
View file

@ -6,6 +6,8 @@
User-agent: LinkChecker
Crawl-delay: 1
Disallow: /secret
# a comment line inside a rule, followed by non-ascii
Disallow: /umlaut/ö
# The following rule should not match
User-agent: *