Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-27 15:24:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
5751 commits 1 branch 0 tags 24 MiB
Commit graph

2804 commits

Author SHA1 Message Date
Bastian Kleineidam
9c5693ad41 Add doc and copyright. 2014-03-30 19:23:42 +02:00
Bastian Kleineidam
4759cee377 Updated mailto: documentation. 2014-03-30 08:30:14 +02:00
Bastian Kleineidam
b6b5c7a12e Simpler link parsing routine. 2014-03-27 19:49:17 +01:00
Bastian Kleineidam
f180592cc4 Increase thread poll intervall to reduce CPU usage. 2014-03-27 17:43:14 +01:00
Bastian Kleineidam
81da2eb48f Code cleanup 2014-03-27 17:19:52 +01:00
Bastian Kleineidam
da0ef8e8ea Fix for moved functions. 2014-03-27 17:19:24 +01:00
Bastian Kleineidam
fa26876f67 Don't use encoding detection since it's very slow. 2014-03-27 12:27:11 +01:00
Bastian Kleineidam
8cf84be2e2 Fix pyopenssl certificate date parsing. 2014-03-26 20:25:44 +01:00
Bastian Kleineidam
49df359317 Some fixes when pyopenssl is used instead of python ssl module. 2014-03-26 19:59:17 +01:00
Bastian Kleineidam
dec0f6c8dc Fix error with SNI checks 2014-03-26 12:38:16 +01:00
Bastian Kleineidam
a8623bc0bc Display SSL info on redirects. 2014-03-26 07:16:03 +01:00
Bastian Kleineidam
be59802569 Set http connection charset. 2014-03-20 21:20:34 +01:00
Bastian Kleineidam
098dede12c Fix warningregex setting in GUI. 2014-03-20 20:46:58 +01:00
Bastian Kleineidam
9cd67dfcb2 More SSL message work. 2014-03-20 20:24:57 +01:00
Bastian Kleineidam
4c76345338 Add certificate valid date info and always set verify flag. 2014-03-19 17:16:42 +01:00
Bastian Kleineidam
9a7ad3a84f Print SSL cipher info for https URLs. 2014-03-19 17:02:34 +01:00
Bastian Kleineidam
931ca4f402 Add missing log keyword arg. 2014-03-19 17:02:00 +01:00
Bastian Kleineidam
71a7898ee6 Don't check non-connected URLs. 2014-03-19 16:33:38 +01:00
Bastian Kleineidam
ce733ae76b Don't check for robots.txt directives in local html files. 2014-03-19 16:33:22 +01:00
Bastian Kleineidam
e528d5f7db Fix ssl connection handling and change plugin type to connection plugin. 2014-03-19 14:28:33 +01:00
Bastian Kleineidam
9be667b52a Do not warn about missing addresses on mailto links that have subjects. 2014-03-18 23:27:59 +01:00
Bastian Kleineidam
2eb6b1b44c Call connect() on unconnected ssl responses. 2014-03-18 23:27:21 +01:00
Bastian Kleineidam
fc73c6ca6e Log number of checked unique URLs. 2014-03-14 23:46:17 +01:00
Bastian Kleineidam
91c6e1d29f Don't log bytes in status. 2014-03-14 22:25:19 +01:00
Bastian Kleineidam
34bdf5c75a Updated copyright and docs. 2014-03-14 22:09:05 +01:00
Bastian Kleineidam
19b8baf08c Move cached queue items to top once in a while. 2014-03-14 22:08:51 +01:00
Bastian Kleineidam
6437f08277 Display downloaded bytes. 2014-03-14 21:06:10 +01:00
Bastian Kleineidam
c51caf1133 Assertions should be earlier. 2014-03-14 20:26:11 +01:00
Bastian Kleineidam
cc401923ac Improve wording of status message. 2014-03-14 20:25:37 +01:00
Bastian Kleineidam
cfff4c4a84 Disable URL length warning for data: URLs. 2014-03-14 20:24:28 +01:00
Bastian Kleineidam
ac78c6d5b8 Internal errors do not stop the checking thread any more. 2014-03-14 20:23:04 +01:00
Bastian Kleineidam
b18854649d Count unique URLs for url queue limit. 2014-03-14 20:21:46 +01:00
Bastian Kleineidam
257644e660 Add cache length function to get number of cached elements. 2014-03-14 20:19:34 +01:00
Bastian Kleineidam
306979abca Add HttpHeaderInfo plugin 2014-03-12 19:28:37 +01:00
Bastian Kleineidam
279db5c5b8 Fix documentation. 2014-03-12 19:22:18 +01:00
Bastian Kleineidam
ccd0d4ead7 Updated the list of unknown or ignored URI schemes. 2014-03-12 19:20:49 +01:00
Bastian Kleineidam
121602df87 Use SSL cert on Windows systems. 2014-03-11 20:58:16 +01:00
Bastian Kleineidam
0ad5969b54 Simplify config dir functions. 2014-03-11 20:23:49 +01:00
Bastian Kleineidam
41d07729bb Install certificate store with installers. 2014-03-10 22:34:37 +01:00
Bastian Kleineidam
ee0717131d Add marker for http debugging 2014-03-10 20:09:05 +01:00
Bastian Kleineidam
9c9cf0c3e2 Check for Python requests >= 2.2.0 2014-03-10 19:31:31 +01:00
Bastian Kleineidam
57edf0923e Updated copyright year 2014-03-10 19:27:22 +01:00
Bastian Kleineidam
bca226c293 Fix assertion checking external links; fix tests 2014-03-10 18:23:44 +01:00
Bastian Kleineidam
40b663cf9e Ignore URLs earlier. 2014-03-10 18:05:11 +01:00
Bastian Kleineidam
6b334dc79b Fix URL result caching. 2014-03-08 19:35:10 +01:00
Bastian Kleineidam
0113f06406 Enable arbitrary output encodings in CSV output. See #467 2014-03-06 22:40:52 +01:00
Bastian Kleineidam
102837b875 Set maximum redirects 2014-03-06 21:58:35 +01:00
Bastian Kleineidam
fab2c2da98 Improve content type setting. 2014-03-05 20:12:19 +01:00
Bastian Kleineidam
ef13a3fce1 Implement sitemap and sitemap index parsing. 2014-03-05 19:26:37 +01:00
Bastian Kleineidam
b72cf252fb Move parseable check down since it might get the content. 2014-03-05 19:26:05 +01:00
Bastian Kleineidam
9ef65cb774 Fix UrlData string representation. 2014-03-05 19:25:40 +01:00
Bastian Kleineidam
00bd549c0c Remove duplicate content type map. 2014-03-05 19:24:58 +01:00
Bastian Kleineidam
380f14453b Fix mimetype guessing from content. 2014-03-05 19:23:58 +01:00
Bastian Kleineidam
192cfab009 Cleanup of the UrlData.is_* functions 2014-03-05 19:23:16 +01:00
Bastian Kleineidam
b17211f162 Set for release. 2014-03-04 21:36:24 +01:00
Bastian Kleineidam
978b24f2d7 Merge branch 'caching' 2014-03-04 07:21:42 +01:00
Bastian Kleineidam
f1076c8813 Increase url-too-long warning. 2014-03-03 23:31:04 +01:00
Bastian Kleineidam
82f81241fd Check all links and add better caching. 2014-03-03 23:29:45 +01:00
Bastian Kleineidam
510af337c1 Improved --version output. 2014-03-01 21:00:16 +01:00
Bastian Kleineidam
74d804ac82 Print release date on --version and internal errors. 2014-03-01 20:59:00 +01:00
Bastian Kleineidam
39df1812c7 Default to 10 threads instead of 100. 2014-03-01 20:49:06 +01:00
Bastian Kleineidam
6f205a2574 Support checking Sitemap: URLs in robots.txt files. 2014-03-01 20:25:19 +01:00
Bastian Kleineidam
0f0d79c7e0 Remove crawl-delay stuff 2014-03-01 20:01:42 +01:00
Bastian Kleineidam
00f8011709 Catch overflowerror in robots.txt crawl-delay 2014-03-01 19:58:22 +01:00
Bastian Kleineidam
0e4d6f6e1a Parse sitemap urls in robots.txt files. 2014-03-01 19:57:57 +01:00
Bastian Kleineidam
78a99717fe Check regular expressions from users for errors. 2014-03-01 19:15:48 +01:00
Bastian Kleineidam
c20005a031 Add missing docstring. 2014-03-01 19:14:43 +01:00
Bastian Kleineidam
39c39b1d9f Disable twill page refresh. 2014-03-01 18:19:29 +01:00
Bastian Kleineidam
0211529d79 Use twill form field number if all else fails. 2014-03-01 18:12:06 +01:00
Bastian Kleineidam
7d84e1e729 Do not check permissions on non-posix systems for now. 2014-03-01 18:01:08 +01:00
Bastian Kleineidam
eb7e52c0e2 -o none sets exit code now 2014-03-01 15:31:39 +01:00
Bastian Kleineidam
f7f5001256 Add missing column name to SQL insert statement. 2014-03-01 12:03:33 +01:00
Bastian Kleineidam
f9bf831804 Remove some empty lines 2014-03-01 12:02:00 +01:00
Bastian Kleineidam
900e04ceda Dynamic language switch in the GUI. 2014-03-01 12:01:47 +01:00
Bastian Kleineidam
9d0255e156 Fix bookmark imports 2014-03-01 10:16:29 +01:00
Bastian Kleineidam
7b34be590b Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements. 2014-03-01 00:12:34 +01:00
Bastian Kleineidam
c806be5c15 Updated copyright 2014-01-08 22:33:04 +01:00
Bastian Kleineidam
c076e312a2 Send an Accept header. 2014-01-08 19:56:00 +01:00
Bastian Kleineidam
f3b435c2a6 Add missing docstrings. 2013-12-24 07:15:31 +01:00
Bastian Kleineidam
e0a2558b2b Updated copyright. 2013-12-24 07:13:16 +01:00
Bastian Kleineidam
845a6a1146 Fix loader in frozen executables. 2013-12-18 20:53:17 +01:00
wummel
9646f0b652 Merge pull request #418 from chuckbjones/reset-url-on-fallback 
Reset to original url when falling back to GET
 2013-12-17 22:37:17 -08:00
Bastian Kleineidam
fbbced4d8f Fix tests 2013-12-13 07:39:59 +01:00
Bastian Kleineidam
5151e68a3e Fix logger config 2013-12-13 07:37:21 +01:00
Bastian Kleineidam
103e00b4d1 Allow disabling of ssl certificate checks. 2013-12-12 22:17:57 +01:00
Bastian Kleineidam
39fb02f9a9 Remember last save result as filetype. 2013-12-12 20:44:09 +01:00
Bastian Kleineidam
5736987b60 Refactor output loggers. 2013-12-11 18:41:55 +01:00
Bastian Kleineidam
78ed1e9e52 Do not GET on POST forms. 2013-12-10 23:42:43 +01:00
Bastian Kleineidam
0ca63797bf Remove content cache. 2013-12-10 23:41:52 +01:00
Bastian Kleineidam
a7c1cdd6f6 Check for help files. 2013-12-10 20:56:26 +01:00
Bastian Kleineidam
2c5ede2eb7 Fallback to GET for Apache Coyote servers. 2013-12-08 08:22:56 +01:00
Bastian Kleineidam
b567f766ba Fix strtime test. 2013-12-06 07:13:44 +01:00
Bastian Kleineidam
6d68e00068 Merge branch 'master' of github.com:wummel/linkchecker 2013-12-04 19:21:45 +01:00
Bastian Kleineidam
023da7c993 Remove the duplicate URL content check. 2013-12-04 19:12:40 +01:00
Bastian Kleineidam
36badddfac Update cookie code from Python module. 2013-12-04 19:05:08 +01:00
wummel
ab54809d95 Merge pull request #426 from alperkokmen/fix-lastmod-format 
Fix ISO formatting for modified datetime.
 2013-12-03 12:22:27 -08:00
Bastian Kleineidam
c676a4c829 Avoid DoS in SSL certificate host matching. 2013-11-30 22:07:23 +01:00
Alper Kokmen
4b3e78cac0 Fix ISO formatting for modified datetime. 
This change will make sure that format_modified returns datetime value
in ISO 8601 format. See W3C documentation at
http://www.w3.org/TR/NOTE-datetime.

Since ```modified``` is parsed and then converted to UTC after it's
extracted from HTTP response, it's safe to assume that format_modified
will always format UTC datetime values.

Instead of ```isoformat``` method which omits timezone information for
UTC values, ```strftime``` with a specific format (that ends with Z)
will be used.
 2013-09-02 15:38:54 -07:00
Charles Jones
4294633c04 Close connection prior to falling back to get, since we change the url back to the original at that time. 2013-08-09 13:08:51 -05:00
Charles Jones
8bc138f18b Reset to original url when falling back to GET 2013-07-30 13:38:59 -05:00