Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-18 19:41:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
7192 commits 1 branch 0 tags 24 MiB
Commit graph

1162 commits

Author SHA1 Message Date
Chris Mayo
c7989eb058 Turn all warnings into errors when building website 
Drop code target, sphinx-autogen is not using conf.py, and since Sphinx
4.0 autosummary is generating stub pages by default anyway.

Mock ctypes and linkcheck.__main__: colorama.py imports MS Windows
libraries, autosummary executes __main__.py.
 2022-11-02 19:24:35 +00:00
LinkChecker
eeef12cbb7 Update man pages 2022-10-31 19:45:56 +00:00
LinkChecker
4246d919df Update doc translation catalogs 2022-10-31 19:45:38 +00:00
Chris Mayo
16bee50068 Move AnchorCheck local file handling into a new class 
When checking local files with AnchorCheck, anchors in URLs
like "example/#anchor" are not supported.

Without AnchorCheck enabled, the Real URL reported for such URLs
was changed to include the anchor when local file checking was added to
AnchorCheck, but it is the directory that is checked.
The same URL was also then used as the Parent URL for the check of each
of the contents of that directory.

For FileUrl this is a revert of:
c221afda ("Enable AnchorCheck to be used with local files", 2022-10-03)
 2022-10-24 19:30:56 +01:00
Chris Mayo
a0eb6d5187 Align documentation of debug in man pages 
Linked to:
b3967f75 ("Correct documentation of --debug in linkchecker(1)", 2022-09-30)
 2022-10-18 19:24:08 +01:00
Chris Mayo
b6eea83f63
Merge pull request #676 from cjmayo/robotmap 
Document sitemaps in linkchecker(1)
 2022-10-17 19:25:57 +01:00
Chris Mayo
93f1d3f4ac Document sitemaps in linkchecker(1) 2022-10-17 19:21:03 +01:00
Chris Mayo
8bc849dfde Make --cookiefile description in linkchecker(1) a bit clearer 2022-10-17 19:21:03 +01:00
Chris Mayo
0c5db040c8 Support maxrequestspersecond less than one 2022-10-05 19:28:01 +01:00
Chris Mayo
e88cf49c8f Enable average HTTP request rate to be above 4 per second 2022-10-05 19:28:01 +01:00
Chris Mayo
354ea933ca
Merge pull request #673 from cjmayo/sitemap 
Fix sitemap output with multiple threads
 2022-10-05 19:20:40 +01:00
Chris Mayo
8b2fb86895 Remove AnchorCheck disabled note in linkcheckerrc(5) 
A partial revert of:
fe6dea12 ("Update documentation for disabled plugins", 2021-11-29)
 2022-10-03 19:33:05 +01:00
Chris Mayo
033dcf89f9
Merge pull request #671 from cjmayo/example 
Fix formatting of ignoreerrors example in linkcheckerrc(5)
 2022-10-03 19:22:36 +01:00
Chris Mayo
b3967f75c4 Correct documentation of --debug in linkchecker(1) 
dns logger was removed in:
e1f72490 ("Move dnspython module into third_party directory.", 2011-05-24)

Threading has not been disabled with --debug since:
eaa8a963 ("Refactor logging configuration.", 2014-05-10)
 2022-09-30 19:22:17 +01:00
Chris Mayo
61071fc5dc
Merge pull request #668 from cjmayo/defaults 
Clarify default values in initial linkcheckerrc and elsewhere
 2022-09-28 19:36:44 +01:00
Chris Mayo
001212b915 Fix formatting of ignoreerrors example in linkcheckerrc(5) 
Introduced in:
8c959589 ("add option to ignore specific errors for specific URLs", 2022-07-21)
 2022-09-28 19:23:04 +01:00
Lukas Pirl
8c959589c3
add option to ignore specific errors for specific URLs 2022-09-25 22:52:04 +02:00
Chris Mayo
4962a302b3 Document default frequency of sitemap logger 2022-09-22 19:24:55 +01:00
Chris Mayo
b8d0928969 Document dialect option of csv logger 2022-09-22 19:24:55 +01:00
Chris Mayo
38dea6b7f4 Fix install with pip git+https 
pip only uses the wheel target.

Save building the metadata twice as a result in GitHub workflows and
update documentation.md.
 2022-09-13 19:32:06 +01:00
Chris Mayo
b6a7f2d313 Don't need hatch to build documentation 
This is a partial revert of:
47d1015e ("Replace setuptools and setup.py with hatch and pyproject.toml", 2022-09-05)

Also hatch is an option to run tests.
 2022-09-13 19:32:06 +01:00
Chris Mayo
af265f3d52 Write all metadata used to _release.py 
Enables running without installing.
Removes use of importlib.metadata.
 2022-09-13 19:32:06 +01:00
Chris Mayo
30e8cfad77
Merge pull request #651 from cjmayo/rate 
Rename url-rate-limited to http-rate-limited
 2022-09-12 19:25:52 +01:00
Chris Mayo
a0b28cc0ff Rename url-rate-limited to http-rate-limited 
Make consistent with the other warnings:

- The first part of the name represents the checker class in which the
  warning is raised

- Update initial comment
 2022-09-06 19:32:24 +01:00
Chris Mayo
fdc4a2eeca Bring development.md up to date 2022-09-05 19:30:38 +01:00
Chris Mayo
728d826eb3 Fix typo makefiles in doc/Makefile 2022-09-05 19:30:38 +01:00
Chris Mayo
931f86848e Remove obsolete doc/dot-travis-osx.yml 2022-09-05 19:30:38 +01:00
Chris Mayo
c627b00755
Merge pull request #639 from cjmayo/hatch 
Replace setuptools and setup.py with hatch and pyproject.toml
 2022-09-05 19:27:48 +01:00
Chris Mayo
47d1015e00 Replace setuptools and setup.py with hatch and pyproject.toml 2022-09-05 19:24:01 +01:00
Chris Mayo
d6936ceb91 Add warning url-content-type-unparseable 2022-09-02 19:29:11 +01:00
Chris Mayo
fb8df6f089 Document URL list in linkchecker(1) 
First enabled in:
b1708fc7 ("get the complete content", 2004-09-03)
 2022-09-02 19:29:11 +01:00
Chris Mayo
f018cc9fd1 Fix typo addtion in linkcheckerrc(5) 2022-09-02 19:17:45 +01:00
Kian-Meng Ang
a70ea9ea14 Fix typos 
Found via `codespell ./linkcheck/ ./tests ./doc/man/en -L bu,noone,fo,pres,shttp`
 2022-09-02 17:20:02 +08:00
Chris Mayo
b35036af2b
Merge pull request #634 from cjmayo/pyxdg 
Remove dependency on pyxdg
 2022-08-30 19:28:03 +01:00
Chris Mayo
fbceca5dc9 Remove dependency on pyxdg 
Read the environment variables and implement the same fallbacks.
Saves a hardly used dependency and is more explicit.
 2022-08-23 19:26:15 +01:00
Chris Mayo
10f3d33041 Finish documenting the use of XDG_CONFIG_HOME and XDG_DATA_HOME 
Introduced by:
a03e2e4a ("use xdg dirs for config & data", 2017-10-17)
 2022-08-23 19:21:53 +01:00
Chris Mayo
3f42367280 Document using the GitHub release artifacts 2022-08-23 19:21:53 +01:00
Chris Mayo
edd4c8443c Document the GitHub workflow for creating man pages and translations PR 2022-08-23 19:21:53 +01:00
Chris Mayo
7099dc73a7 Sphinx 6.0 extlinks compatibility 
WARNING: extlinks: Sphinx-6.0 will require a caption string to
contain exactly one '%s' and all other '%' need to be escaped as '%%'.
 2022-08-23 19:21:53 +01:00
Chris Mayo
48a1dde27a Document url-rate-limited 
Introduced in:
dcdc64e8 ("Turn status code 429 into warning instead of failure",
2020-03-25)
 2022-07-11 19:27:07 +01:00
Chris Mayo
a67e8fd0f5 Test and recommend pdfminer.six for PdfParser 
pdfminer is no longer actively maintained; distributions are now
packaging pdfminer.six as pdfminer.
 2022-05-18 19:29:54 +01:00
Chris Mayo
34ba737710 Remove cchardet from Docker image, tox env and recommendation 
Beautiful Soup 4.11 will use charset-normalizer.
 2022-04-11 19:26:18 +01:00
Chris Mayo
7673cd6c69 Update files to be edited during release process 2022-02-22 19:35:04 +00:00
Chris Mayo
b92418a950 Use R for Requests in install.txt 2022-02-22 19:35:04 +00:00
Chris Mayo
db012a33cf Update documentation.md with PKG-INFO 
_LinkChecker_configdata.py not created since:
fb03843b ("Remove _LinkChecker_configdata.py", 2021-12-30)
 2022-02-22 19:35:04 +00:00
Chris Mayo
5c0d66dd74 Raise minimum Python requirement to 3.7 2021-12-30 19:27:04 +00:00
Chris Mayo
169305a171 Update change log and upgrading for 10.1.0 2021-12-22 19:29:56 +00:00
LinkChecker
8b61794ba9 Update man pages 2021-12-21 19:41:24 +00:00
LinkChecker
41cb8880ba Update doc translation catalogs 2021-12-21 19:41:06 +00:00
Chris Mayo
58adec0a28
Merge pull request #599 from cjmayo/sitemap 
Use sphinx-sitemap to create a sitemap
 2021-12-20 19:25:41 +00:00
Chris Mayo
abd5a68c27 Use sphinx-sitemap to create a sitemap 2021-12-17 19:25:43 +00:00
Chris Mayo
de3361fae2 More proxy documentation updates 
a60648e34 ("Remove support for ftp_proxy", 2021-12-13)
9504a6ddd ("Document the curl_ca_bundle environment variable", 2021-12-13)
 2021-12-17 19:22:20 +00:00
Chris Mayo
c28559e0d9 Recommend ensurepip instead of get-pip.py 
Avoids manual download of the get-pip.py file.
 2021-12-15 19:34:24 +00:00
LinkChecker
7497f1aa63 Update man pages 2021-12-14 19:27:40 +00:00
LinkChecker
289ef5dc5b Update doc locale 2021-12-14 19:27:20 +00:00
Chris Mayo
315e729ec3 Fix typo "variables" in linkchecker(1) Proxy Support 2021-12-13 19:25:23 +00:00
Chris Mayo
9504a6dddf Document the curl_ca_bundle environment variable 2021-12-13 19:25:23 +00:00
Chris Mayo
a2e379a595 Remove built-in GNOME and KDE proxy support 
Only http_proxy was ever supported.

Requests uses urllib.request.getproxies().

Fedora 35 and Ubuntu 20.04 do set proxy environment variables when
settings are added through the GUI.

GNOME location of proxy settings is subject to change:
https://wiki.gnome.org/Projects/NetworkManager/Proxies
https://gitlab.gnome.org/GNOME/gsettings-desktop-schemas/-/issues/27
 2021-12-13 19:25:23 +00:00
Chris Mayo
fe5a34c68f Remove linkcheck.checker.proxysupport 
Set up the requests.Session() with the complete proxy configuration
to fix a problem with using an HTTP server as an HTTPS proxy and
potential redirection issues.

Requests handles no_proxy.
 2021-12-13 19:25:23 +00:00
Chris Mayo
a60648e348 Remove support for ftp_proxy 
Was limited to HTTP proxy servers and prevents simplifying and fixing
HTTP proxy support.
 2021-12-13 19:25:23 +00:00
Chris Mayo
2cc1319bee GitHub release must be created before Python distribution files 
setuptools_scm will use the release tag.
 2021-12-06 19:40:14 +00:00
Chris Mayo
d2d2a563cc Make installing from the git archive of a tagged commit possible 
aka make GitHub release assets usable.

Requires setuptools-scm-git-archive.
 2021-12-06 19:40:02 +00:00
Chris Mayo
3b19680e97 Add guidance on character set detecting including cchardet 2021-12-06 19:24:26 +00:00
Chris Mayo
6269134017 Correct documentation of text output encoding 
50ea41b12 ("use preferred locale for default encoding", 2005-10-13)
714147cb2 ("Improved language and encoding detection by using local.getdefaultlocale() instead of locale.getlocale(category=LC_ALL)", 2009-03-04)
 2021-11-29 19:48:50 +00:00
Chris Mayo
6de93650f9 Group output documentation 2021-11-29 19:48:50 +00:00
Chris Mayo
a4b14047d6 Make quiet/-q set application logging to warning 2021-11-29 19:48:50 +00:00
Chris Mayo
eb33b1c1e5 Remove --warning-regex documentation 
Moved to RegexCheck plugin:
7b34be590 ("Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements.", 2014-03-01)
 2021-11-29 19:48:50 +00:00
Chris Mayo
a7155db64a Correct documentation of CSV separator 
851e1121e ("Use semicolon as default CSV separator.", 2010-07-31)
 2021-11-29 19:48:50 +00:00
Chris Mayo
895e2ddb43 Improve documentation of output 2021-11-29 19:48:50 +00:00
Chris Mayo
fe6dea12de Update documentation for disabled plugins 2021-11-29 19:35:34 +00:00
Chris Mayo
af93386bcc Version application using setuptools_scm 
Version has the format X.Y.Z.postN+g12345678
 2021-11-28 18:55:46 +00:00
Chris Mayo
e297b1a477 Stop including binary translation catalogs in the source 
Makes it easier to contribute and accept translations.

Does add a new dependency on installation from source, polib. This is
not made mandatory because not all users will want translations.

polib has no further dependencies and can be installed anywhere
LinkChecker can, using pip too.

Add release process step to check polib is installed

The Wheel still includes translations. Recommend install from PyPI in
the README.

Translations were being compiled in setup.py until 2014, using a bundled
copy of msgfmt.py.
9c3739f1c ("Replace msgfmt.py with local tools.", 2014-09-08)
 2021-11-22 19:30:33 +00:00
Chris Mayo
1194f5d943 Update pip URL in install.txt 2021-11-22 19:30:33 +00:00
Chris Mayo
d99e600edd Add translation .po file checks 2021-11-22 19:30:33 +00:00
Chris Mayo
119f6a961e Use an HTML list for custom menu entries in documentation 
For compatibility with sphinx_rtd_theme 1.0, which otherwise renders the
entries with buttons.
 2021-08-22 19:26:03 +01:00
Chris Mayo
49d994d7cc Reference GitHub Actions instead of Travis in documentation 2021-08-22 19:26:03 +01:00
Chris Mayo
b9b4936729 Fix broken Edit on GitHub links in documentation 2021-08-12 19:28:50 +01:00
Paul Haerle
f395c74aac
Make ResultCache max_size configurable (#544) 
* Make ResultCache max_size configurable

fixes #463

* Add tests and docs.

* fix documentation...

...adapt the source, not the auto-generated man pages themselves as
requested in #544.

* fix typo.
 2021-06-21 19:45:19 +01:00
Chris Mayo
c4b8a4aeec Changes for release 10.0.1 2021-01-28 19:20:24 +00:00
Chris Mayo
b84aac8de9 Update copyright in man pages 2021-01-28 19:20:24 +00:00
Chris Mayo
e7e9c3a865 Update release process 2021-01-28 19:20:24 +00:00
Chris Mayo
732f988500 Changes for release 10.0.0 2021-01-14 19:48:22 +00:00
Chris Mayo
4132f245d4 Promote links to the man pages on the home page 2021-01-14 19:48:22 +00:00
Chris Mayo
4237a0be76 Update documentation of gh-pages branch 2021-01-14 19:48:22 +00:00
Chris Mayo
1adcea1ff4
Merge branch 'master' into edit-index 2020-10-15 20:02:29 +01:00
Chris Mayo
5ba15f0e0e
Merge branch 'master' into biplist 2020-10-15 19:54:40 +01:00
Chris Mayo
b106723685 Add setup.py build step to documentation 2020-10-15 19:35:24 +01:00
Chris Mayo
ffb1d9953c Fix the edit link for the web site front page 2020-10-14 19:28:18 +01:00
Chris Mayo
a5706e233b Publish documentation to the root of gh-pages 2020-10-14 19:22:59 +01:00
Chris Mayo
a585925d43 Automatically publish documentation on GitHub Pages 2020-10-13 19:28:48 +01:00
Chris Mayo
e922dd0224 Stop using biplist 
plistlib has supported binary files since Python 3.4.
 2020-10-12 19:55:46 +01:00
Chris Mayo
6fa48359a7
Merge branch 'master' into changes 2020-10-03 17:27:30 +01:00
Chris Mayo
590dc66a67 Update changelog 2020-10-02 19:12:06 +01:00
Chris Mayo
8bd6584c85 Correct documentation of aborttimeout, no matching option 
$ grep -rn aborttimeout linkcheck
linkcheck/configuration/confparse.py:167:        self.read_int_option(section, "aborttimeout", min=1)
linkcheck/configuration/__init__.py:184:        self["aborttimeout"] = 300
linkcheck/director/aggregator.py:173:            timeout=strformat.strduration_long(self.config["aborttimeout"]),
linkcheck/director/aggregator.py:200:        timeout = self.config["aborttimeout"]

config["aborttimeout"] is only set from aborttimeout and is the only
value used by Aggregate.abort().
 2020-10-02 19:04:40 +01:00
Chris Mayo
319993bb77 Update missed multiline hyperlinks in German linkcheckerrc(5) 
Missed from:
1f600ae1 ("Update German man pages", 2020-09-25)
 2020-10-02 19:04:40 +01:00
Chris Mayo
1f600ae1c8 Update German man pages 2020-09-25 19:33:15 +01:00
Chris Mayo
26a6ff9259 Run make -C doc locale 2020-09-25 19:33:15 +01:00
Chris Mayo
e875a5f9e5 Assorted German man page translation fixes 2020-09-25 19:33:15 +01:00
Chris Mayo
ee9e00090f Update English man pages because of multiline hyperlinks 2020-09-25 19:33:15 +01:00
Chris Mayo
54e34b2774
Merge pull request #490 from cjmayo/dnspython-2 
Fix deprecation warning for resolver.query(), Drop Python 3.5 and add 3.9
 2020-09-23 19:52:34 +01:00