Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-13 17:13:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
5933 commits 1 branch 0 tags 24 MiB
Commit graph

2848 commits

Author SHA1 Message Date
Bastian Kleineidam
c8df9355f0 Try to use the SSL certs from the certifi package. 2014-09-05 20:00:30 +02:00
Bastian Kleineidam
c684918ba6 Ignore urllib3 warnings about invalid SSL certs since we check them ourselves. 2014-09-05 20:00:00 +02:00
Bastian Kleineidam
2354f16dbb Catch urllib3 errors. 2014-09-05 19:59:28 +02:00
Arlo Louis O'Keeffe
52337f82cb Use correct attribute 2014-09-03 09:36:22 +02:00
Bastian Kleineidam
85dadc1f1a Add documentation 2014-07-16 07:37:19 +02:00
Bastian Kleineidam
37664ea8a4 Fix Word file check plugin. 2014-07-15 22:39:41 +02:00
Bastian Kleineidam
b646293fd6 Remove unused import. 2014-07-15 22:38:57 +02:00
Bastian Kleineidam
29193bbcc9 Fix login URL cookies and don't sanitize after config reading. 2014-07-15 22:23:38 +02:00
Bastian Kleineidam
032c4091c3 Some easy python3 compatibility changes. 2014-07-15 18:40:47 +02:00
Bastian Kleineidam
90257a1b5e Replace twill with custom code. 2014-07-15 18:37:05 +02:00
Bastian Kleineidam
a665d35feb Use proxies and checker session in robots.txt. 2014-07-14 20:28:28 +02:00
Bastian Kleineidam
266e9e189f Further code cleanup. 2014-07-14 20:14:00 +02:00
Bastian Kleineidam
6c38b4165a Use given HTTP auth data for robots.txt fetching. 2014-07-14 19:50:11 +02:00
Bastian Kleineidam
7838521b6e Code cleanup. 2014-07-14 19:49:01 +02:00
Bastian Kleineidam
100ce11d40 Sanitize CGI configuration. 2014-07-13 21:56:01 +02:00
Bastian Kleineidam
eafa1ed2da Updated unknown URL schemes. 2014-07-13 21:51:53 +02:00
Bastian Kleineidam
176b95a30e Do not strip quotes from resolved URLs. 2014-07-11 00:43:46 +02:00
Bastian Kleineidam
27702ddbac Catch log output start errors. 2014-07-09 21:54:47 +02:00
Bastian Kleineidam
6ff89e9e8c Fix GUI startup 2014-07-06 20:20:03 +02:00
Bastian Kleineidam
0fa7ed2699 Fix empty URL handling. 2014-07-03 23:34:40 +02:00
Bastian Kleineidam
1590ab6240 cleanup 2014-07-01 21:12:47 +02:00
Bastian Kleineidam
9a124513e3 Merge branch 'master' of github.com:wummel/linkchecker 2014-07-01 21:11:33 +02:00
wummel
9bb3852edf Merge pull request #515 from Mark-Hetherington/extern-redirect 
When following redirections update url.extern
 2014-07-01 21:11:13 +02:00
Bastian Kleineidam
12cc12db53 Add get_redirects() function. 2014-07-01 21:11:06 +02:00
Bastian Kleineidam
cde261c009 Parse Refresh: and Content-Location: header values for URLs. 2014-07-01 20:16:43 +02:00
Bastian Kleineidam
c3ec91ac6d Fix intern URL search pattern. 2014-06-13 23:52:21 +02:00
Bastian Kleineidam
ad8eb424f3 Merge Mark-Hetherington-xml-parse-warn with slight modifications. 2014-06-13 20:50:37 +02:00
Mark Hetherington
34d83db29c When following redirections update url.extern 2014-05-19 14:59:58 +10:00
Bastian Kleineidam
eaa8a963ec Refactor logging configuration. 2014-05-10 21:23:06 +02:00
Bastian Kleineidam
4b28e6e860 Move mime stuff into own submodule. 2014-05-10 21:22:10 +02:00
Bastian Kleineidam
9b794b936c Print interrupt note in text output. 2014-04-30 20:17:33 +02:00
Bastian Kleineidam
43c2e6641b Logging refactor, interrupt and abort flags added. 2014-04-30 19:59:43 +02:00
Bastian Kleineidam
b152ce7a6e Add PDF test and fix page number. 2014-04-29 18:53:24 +02:00
Bastian Kleineidam
0d9881cf03 Fix add_url() with local files. 2014-04-29 18:43:21 +02:00
Bastian Kleineidam
82dd76b0d7 Add PDF link parsing. 2014-04-28 18:13:45 +02:00
Bastian Kleineidam
0ffdea2b8d Added parser plugins and the applies_to() function. 2014-04-28 18:11:19 +02:00
Bastian Kleineidam
0f8ee234c3 Fix documentation. 2014-04-28 18:10:20 +02:00
Bastian Kleineidam
6bae3e0f49 Use the same request arguments for redirects. 2014-04-23 22:03:44 +02:00
Bastian Kleineidam
981079c041 Support itemtype attribute parsing. 2014-04-23 22:03:20 +02:00
Bastian Kleineidam
4232b69633 Support <img> srcset attribute parsing. 2014-04-10 17:51:59 +02:00
Bastian Kleineidam
6caf654031 Parse Link: heaaders. 2014-04-10 17:50:55 +02:00
Bastian Kleineidam
22caa9367a Refactor recursion checks. 2014-04-10 17:50:55 +02:00
Bastian Kleineidam
08fbd891ef Do not check external robots.txt sitemaps. 2014-04-09 19:44:29 +02:00
Bastian Kleineidam
c57f607fc3 Use urldata.add_url() 2014-04-07 18:54:33 +02:00
Bastian Kleineidam
9c5693ad41 Add doc and copyright. 2014-03-30 19:23:42 +02:00
Bastian Kleineidam
4759cee377 Updated mailto: documentation. 2014-03-30 08:30:14 +02:00
Bastian Kleineidam
b6b5c7a12e Simpler link parsing routine. 2014-03-27 19:49:17 +01:00
Bastian Kleineidam
f180592cc4 Increase thread poll intervall to reduce CPU usage. 2014-03-27 17:43:14 +01:00
Bastian Kleineidam
81da2eb48f Code cleanup 2014-03-27 17:19:52 +01:00
Bastian Kleineidam
da0ef8e8ea Fix for moved functions. 2014-03-27 17:19:24 +01:00
Bastian Kleineidam
fa26876f67 Don't use encoding detection since it's very slow. 2014-03-27 12:27:11 +01:00
Bastian Kleineidam
8cf84be2e2 Fix pyopenssl certificate date parsing. 2014-03-26 20:25:44 +01:00
Bastian Kleineidam
49df359317 Some fixes when pyopenssl is used instead of python ssl module. 2014-03-26 19:59:17 +01:00
Bastian Kleineidam
dec0f6c8dc Fix error with SNI checks 2014-03-26 12:38:16 +01:00
Bastian Kleineidam
a8623bc0bc Display SSL info on redirects. 2014-03-26 07:16:03 +01:00
Bastian Kleineidam
be59802569 Set http connection charset. 2014-03-20 21:20:34 +01:00
Bastian Kleineidam
098dede12c Fix warningregex setting in GUI. 2014-03-20 20:46:58 +01:00
Bastian Kleineidam
9cd67dfcb2 More SSL message work. 2014-03-20 20:24:57 +01:00
Bastian Kleineidam
4c76345338 Add certificate valid date info and always set verify flag. 2014-03-19 17:16:42 +01:00
Bastian Kleineidam
9a7ad3a84f Print SSL cipher info for https URLs. 2014-03-19 17:02:34 +01:00
Bastian Kleineidam
931ca4f402 Add missing log keyword arg. 2014-03-19 17:02:00 +01:00
Bastian Kleineidam
71a7898ee6 Don't check non-connected URLs. 2014-03-19 16:33:38 +01:00
Bastian Kleineidam
ce733ae76b Don't check for robots.txt directives in local html files. 2014-03-19 16:33:22 +01:00
Bastian Kleineidam
e528d5f7db Fix ssl connection handling and change plugin type to connection plugin. 2014-03-19 14:28:33 +01:00
Bastian Kleineidam
9be667b52a Do not warn about missing addresses on mailto links that have subjects. 2014-03-18 23:27:59 +01:00
Bastian Kleineidam
2eb6b1b44c Call connect() on unconnected ssl responses. 2014-03-18 23:27:21 +01:00
Bastian Kleineidam
fc73c6ca6e Log number of checked unique URLs. 2014-03-14 23:46:17 +01:00
Bastian Kleineidam
91c6e1d29f Don't log bytes in status. 2014-03-14 22:25:19 +01:00
Bastian Kleineidam
34bdf5c75a Updated copyright and docs. 2014-03-14 22:09:05 +01:00
Bastian Kleineidam
19b8baf08c Move cached queue items to top once in a while. 2014-03-14 22:08:51 +01:00
Bastian Kleineidam
6437f08277 Display downloaded bytes. 2014-03-14 21:06:10 +01:00
Bastian Kleineidam
c51caf1133 Assertions should be earlier. 2014-03-14 20:26:11 +01:00
Bastian Kleineidam
cc401923ac Improve wording of status message. 2014-03-14 20:25:37 +01:00
Bastian Kleineidam
cfff4c4a84 Disable URL length warning for data: URLs. 2014-03-14 20:24:28 +01:00
Bastian Kleineidam
ac78c6d5b8 Internal errors do not stop the checking thread any more. 2014-03-14 20:23:04 +01:00
Bastian Kleineidam
b18854649d Count unique URLs for url queue limit. 2014-03-14 20:21:46 +01:00
Bastian Kleineidam
257644e660 Add cache length function to get number of cached elements. 2014-03-14 20:19:34 +01:00
Bastian Kleineidam
306979abca Add HttpHeaderInfo plugin 2014-03-12 19:28:37 +01:00
Bastian Kleineidam
279db5c5b8 Fix documentation. 2014-03-12 19:22:18 +01:00
Bastian Kleineidam
ccd0d4ead7 Updated the list of unknown or ignored URI schemes. 2014-03-12 19:20:49 +01:00
Bastian Kleineidam
121602df87 Use SSL cert on Windows systems. 2014-03-11 20:58:16 +01:00
Bastian Kleineidam
0ad5969b54 Simplify config dir functions. 2014-03-11 20:23:49 +01:00
Bastian Kleineidam
41d07729bb Install certificate store with installers. 2014-03-10 22:34:37 +01:00
Bastian Kleineidam
ee0717131d Add marker for http debugging 2014-03-10 20:09:05 +01:00
Bastian Kleineidam
9c9cf0c3e2 Check for Python requests >= 2.2.0 2014-03-10 19:31:31 +01:00
Bastian Kleineidam
57edf0923e Updated copyright year 2014-03-10 19:27:22 +01:00
Bastian Kleineidam
bca226c293 Fix assertion checking external links; fix tests 2014-03-10 18:23:44 +01:00
Bastian Kleineidam
40b663cf9e Ignore URLs earlier. 2014-03-10 18:05:11 +01:00
Bastian Kleineidam
6b334dc79b Fix URL result caching. 2014-03-08 19:35:10 +01:00
Bastian Kleineidam
0113f06406 Enable arbitrary output encodings in CSV output. See #467 2014-03-06 22:40:52 +01:00
Bastian Kleineidam
102837b875 Set maximum redirects 2014-03-06 21:58:35 +01:00
Bastian Kleineidam
fab2c2da98 Improve content type setting. 2014-03-05 20:12:19 +01:00
Bastian Kleineidam
ef13a3fce1 Implement sitemap and sitemap index parsing. 2014-03-05 19:26:37 +01:00
Bastian Kleineidam
b72cf252fb Move parseable check down since it might get the content. 2014-03-05 19:26:05 +01:00
Bastian Kleineidam
9ef65cb774 Fix UrlData string representation. 2014-03-05 19:25:40 +01:00
Bastian Kleineidam
00bd549c0c Remove duplicate content type map. 2014-03-05 19:24:58 +01:00
Bastian Kleineidam
380f14453b Fix mimetype guessing from content. 2014-03-05 19:23:58 +01:00
Bastian Kleineidam
192cfab009 Cleanup of the UrlData.is_* functions 2014-03-05 19:23:16 +01:00
Bastian Kleineidam
b17211f162 Set for release. 2014-03-04 21:36:24 +01:00
Bastian Kleineidam
978b24f2d7 Merge branch 'caching' 2014-03-04 07:21:42 +01:00