2 commits

Author	SHA1	Message	Date
Chris Mayo	1d1d9c3bde	Add testing for variants of the robots meta directive	2020-04-29 20:14:10 +01:00
Chris Mayo	56b8c9f7ab	Add tests for <meta name="robots" content="nofollow"> ... norobots.html was used for testing <meta name="robots" content="nofollow"> in local files until [1]. This commit reinstates local file testing and adds an http test. Checking is reported by checker.httpurl.HttpUrl.content_allows_robots(). [1] ce733ae7 ("Don't check for robots.txt directives in local html files.", 2014-03-19)	2020-04-18 20:30:46 +01:00