Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-05 15:30:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
6271 commits 1 branch 0 tags 24 MiB
Commit graph

6271 commits

This branch
This branch
All branches
Author SHA1 Message Date
Chris Mayo
08ddf658bc
Merge pull request #366 from cjmayo/userorpwd 
Support login forms with user and/or password
 2020-05-13 19:37:44 +01:00
Chris Mayo
736c893707
Merge pull request #377 from cjmayo/tidyten3 
Remove u string prefixes
 2020-05-13 19:36:54 +01:00
Chris Mayo
00c4a30386 Add user and password only loginurl tests 2020-05-13 19:32:29 +01:00
Chris Mayo
3ace021264 Support login forms with user and/or password 2020-05-13 19:32:25 +01:00
Chris Mayo
31a9f68c46
Merge pull request #367 from cjmayo/loginurl 
Add test for loginurl
 2020-05-12 20:08:57 +01:00
anarcat
8d41f4a86b
Merge pull request #376 from cjmayo/tidyten2 
Remove # -*- coding: lines and inheriting object
 2020-05-11 13:29:04 -04:00
Marius Gedminas
f82e10cd39
Merge pull request #379 from dcycle/python3 
Update README and Dockerfile with Python 3
 2020-05-10 12:15:57 +03:00
alberto56
1ebc6a1431 use python3 in readme and dockerfile 2020-05-09 08:03:23 -04:00
Chris Mayo
44e81d27dd Remove inheriting object 
All Python 3 classes are new-style.
 2020-05-08 10:45:31 +01:00
Chris Mayo
b0ea72e8c1 Remove # -*- coding: lines 
Except for tests that include non-unicode characters:

tests/test_po.py
tests/test_strformat.py
tests/test_url.py
tests/checker/test_error.py
tests/checker/test_news.py
 2020-05-08 10:45:31 +01:00
Marius Gedminas
fdbb3a3b76
Merge pull request #378 from linkchecker/fix-metaclass 
Make _Logger an abstract base class
 2020-05-02 17:34:36 +03:00
Marius Gedminas
22b0165b72 Make _Logger an abstract base class 
The __metaclass__ syntax is a Python-2-ism.  It was replaced with

    class _Logger (object, metaclass=abc.ABCMeta):

in Python 3.  And then Python 3.4 introduced abc.ABC which is an empty
class that has ABCMeta as the metaclass, making it simpler to define
abstract base classes.
 2020-04-30 23:09:42 +03:00
Chris Mayo
4d3e5abcfa Remove u string prefixes 2020-04-30 20:11:59 +01:00
anarcat
ab476fa4bf
Merge pull request #364 from cjmayo/parser5 
Stop using HTML handlers and improve login form error handling
 2020-04-30 09:28:48 -04:00
anarcat
19d683bca5
Merge pull request #375 from cjmayo/parser5a 
Improve login form handling
 2020-04-30 09:26:21 -04:00
Chris Mayo
1d1d9c3bde Add testing for variants of the robots meta directive 2020-04-29 20:14:10 +01:00
Chris Mayo
12a948894b Fix space style in linkcheck/htmlutil/linkparse.py 2020-04-29 20:07:00 +01:00
Chris Mayo
9eed070a73 Stop using HTML handlers 
LinkFinder is the only remaining HTML handler therefore no need for
htmlsoup.process_soup() as an independent function or TagFinder as a
base class.
 2020-04-29 20:07:00 +01:00
Chris Mayo
a1433767e5 Replace HtmlPrettyPrinter with pretty_print_html() 2020-04-29 20:07:00 +01:00
Chris Mayo
0361d9e0e8 Remove encoding and default fd from HtmlPrettyPrinter 
Neither are used.
 2020-04-29 20:07:00 +01:00
Chris Mayo
4ffdbf2406 Replace MetaRobotsFinder using BeautifulSoup.find() 2020-04-29 20:07:00 +01:00
Chris Mayo
a51f02cf66 Improve error handling and debugging for login form 2020-04-27 18:06:29 +01:00
Chris Mayo
9a33c2a659 Make requesting login form password work on Python 3 2020-04-27 18:06:29 +01:00
Chris Mayo
8fc0dcc055 Make matching login form credentials case-sensitive 
The keys of the form.data dictionary are case-sensitive and therefore a
KeyError was possible if the configured values are not identical to
the input element name attributes.
 2020-04-27 18:06:29 +01:00
Chris Mayo
7a6ef938cc Rename htmlutil.formsearch to htmlutil.loginformsearch 
Make it clear that this module has only one specific use.
 2020-04-27 18:06:29 +01:00
anarcat
350f8bfef9
Merge pull request #373 from linkchecker/fix-swf-parsing 
SWF files are binary data
 2020-04-27 09:39:52 -04:00
Marius Gedminas
680783b1ff SWF files are binary data 
Should fix #372.
 2020-04-27 11:25:37 +03:00
anarcat
183d483074
Merge pull request #365 from cjmayo/tidyten1 
Remove use of the future package
 2020-04-26 12:02:30 -04:00
anarcat
125146fb2c
Merge pull request #361 from cjmayo/parser4 
Rename htmlsax.py to htmlsoup.py and add test_content_allows_robots
 2020-04-25 17:56:29 -04:00
anarcat
87079312db
Merge pull request #371 from cjmayo/manhtml 
Switch to mandoc for generating html man pages
 2020-04-24 18:59:10 -04:00
Chris Mayo
b7c8ad9be7 Fix typo for -Dplugin in man page 2020-04-24 19:46:30 +01:00
Chris Mayo
5dd448cf05 Add link to unknownurl.py in man page 2020-04-24 19:46:30 +01:00
Chris Mayo
a506800c07 Replace `` in man page with bold formatting 2020-04-24 19:46:30 +01:00
Chris Mayo
e3b77f810e Update external links in man pages to https 2020-04-24 19:46:30 +01:00
Chris Mayo
a205a3722b Update man pages to optimise for both html and man 
- Use "LinkChecker User Manual" as the source for both pages.
- .UR/.UE for external links to allow mandoc to create links in html.
- Use Linux man-pages format for cross references e.g.
  .BR linkcheckerrc (5) which are replace in the html by the Makefile.
 2020-04-24 19:46:30 +01:00
Chris Mayo
441cda5e15 Switch to mandoc for generating html man pages 
Removes the need for diff files and is a currently maintained project.

Cross references are only supported for mdoc macros but because we only
have two pages this can be achieved with sed.

A clean target is added to the Makefile to make development easier.
 2020-04-24 19:46:30 +01:00
Chris Mayo
3b8af403be Add test for loginurl 
A new cgi-bin directory is created to identify the scripts to be run by
http.server.CGIHTTPRequestHandler.
 2020-04-19 19:05:55 +01:00
Chris Mayo
56b8c9f7ab Add tests for <meta name="robots" content="nofollow"> 
norobots.html was used for testing <meta name="robots"
content="nofollow"> in local files until [1]. This commit reinstates
local file testing and adds an http test.

Checking is reported by checker.httpurl.HttpUrl.content_allows_robots().

[1] ce733ae7 ("Don't check for robots.txt directives in local html
files.", 2014-03-19)
 2020-04-18 20:30:46 +01:00
Chris Mayo
d189445a8e LinkFinder does not raise StopParse 2020-04-18 20:30:46 +01:00
Chris Mayo
ee6628a831 Move HtmlParser/htmlsax.py to htmlutil/htmlsoup.py 
Remove one subpackage and some import lines where htmlutil.linkparse is
also being used.
 2020-04-18 20:30:45 +01:00
anarcat
0f18c9b8f0
Merge pull request #360 from cjmayo/parser3 
Replace Parser class using BeautifulSoup.find_all()
 2020-04-18 14:37:03 -04:00
Chris Mayo
384e1e196d Remove Python 2 gettext builtin installation 2020-04-15 19:49:16 +01:00
Chris Mayo
a83fbb56c0 Remove from __future__ imports 2020-04-15 19:49:16 +01:00
Chris Mayo
f5e7f3a382 Remove use of the future package 
It was providing Python 2 compatibility.
 2020-04-15 19:49:16 +01:00
Chris Mayo
0795e3c1b4 Replace Parser class using BeautifulSoup.find_all() 2020-04-10 13:51:09 +01:00
Chris Mayo
eb3cf28baa Remove support for start_end_element() callback 
The LinkFinder handler start_end_element() callback does nothing apart
from call start_element().
 2020-04-10 13:51:09 +01:00
Chris Mayo
c9f17e92b9 Remove support for end_element() callback 2020-04-10 13:51:09 +01:00
Chris Mayo
48b590cf8b Replace FormFinder using BeautifulSoup.find_all() 
FormFinder was the only handler that used an end_element() callback and
was therefore a blocker to moving the Parser class to use
BeautifulSoup.find_all()

FormFinder was a specialised handler used to parse a login form at
the start of a session if the user had configured authentication
credentials.
 2020-04-10 13:51:05 +01:00
anarcat
d80a075372
Merge pull request #357 from cjmayo/parser2 
Simplify the Parser class
 2020-04-09 15:22:14 -04:00
Chris Mayo
974915cc4f Remove encoding from Parser 
Only used by the test and an attribute of the soup object.
 2020-04-08 20:03:35 +01:00