Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-16 18:41:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
5618 commits 1 branch 0 tags 24 MiB
Commit graph

771 commits

Author SHA1 Message Date
Bastian Kleineidam
033280cfb9 Remove workarounds for old Python versions. 2012-01-04 20:17:53 +01:00
Bastian Kleineidam
3d9958dfbb Parse Safari bookmark files. 2011-12-17 16:38:25 +01:00
Bastian Kleineidam
a2978209e6 Ignore errors trying to get FTP feature set. 2011-10-18 13:10:49 +02:00
Bastian Kleineidam
27b7b1cb49 Fix W3C HTML validation. 2011-10-09 21:16:45 +02:00
Bastian Kleineidam
89ec0ee6a1 Check multiple matches of warning regex. 2011-10-09 19:00:35 +02:00
Bastian Kleineidam
09d9264470 Updated copyright. 2011-08-04 20:40:49 +02:00
Bastian Kleineidam
cdf91a0321 Improve cookie info message and fix cookie test cases. 2011-08-04 18:34:56 +02:00
Bastian Kleineidam
48413de418 Display warning message for each cookie parsing error. 2011-08-03 19:27:36 +02:00
Bastian Kleineidam
c99b75899d Send multiple cookie values in one header. 2011-08-02 21:57:16 +02:00
Bastian Kleineidam
c70bd68ef1 Refactor sending of cookie data in client into separate function. 2011-08-02 20:45:26 +02:00
Bastian Kleineidam
51bcccfdfe Added new option --user-agent to set the User-Agent header. 2011-07-25 21:09:49 +02:00
Bastian Kleineidam
552c71a3ca Do not append a stray newline character when encoding authentication information to base64. 2011-07-25 20:02:01 +02:00
Bastian Kleineidam
2550e16040 Remove query part from file links. 2011-05-29 17:49:01 +02:00
Bastian Kleineidam
5515645af6 Reset content type setting after loading HTTP headers. 2011-05-28 17:59:44 +02:00
Bastian Kleineidam
0f70438a87 Updated copyright. 2011-05-28 08:44:21 +02:00
Bastian Kleineidam
684a9b5bf6 Add includes to dns.rdtypes.IN/ANY in setup.py, not in mailtourl.py module. 2011-05-25 21:03:10 +02:00
Bastian Kleineidam
e1f724908d Move dnspython module into third_party directory. 2011-05-24 20:18:58 +02:00
Bastian Kleineidam
72b65d94df Only check anchors in HTML pages. 2011-05-22 17:33:16 +02:00
Bastian Kleineidam
e5c2271533 Only check warning patterns in parseable contents. 2011-05-22 17:32:26 +02:00
Bastian Kleineidam
68ea03ee16 Support both Chromium and Google Chrome profile dirs to find bookmark files. 2011-05-21 11:47:54 +02:00
Bastian Kleineidam
78790d7c8d Improved anchor warning message display. 2011-05-20 06:48:06 +02:00
Bastian Kleineidam
03feaeca91 Correct warning about unparsable cookies. 2011-05-18 20:56:31 +02:00
Bastian Kleineidam
343cf9703d Code cleanup: indentation, unused variables etc. 2011-05-15 18:36:30 +02:00
Bastian Kleineidam
a1f0867c74 Updated copyright 2011-05-06 20:27:36 +02:00
Bastian Kleineidam
10bbb696e8 Limit download file size to 5MB. 2011-05-05 21:10:55 +02:00
Bastian Kleineidam
1f9cd2f67f Redirection refactoring part 2 of 2. 2011-04-27 13:33:01 +02:00
Bastian Kleineidam
dd53c78096 Redirection refactoring part 1. 2011-04-27 12:02:30 +02:00
Bastian Kleineidam
f566f98fe5 Allow redirections for URLs given by the user. 2011-04-27 11:21:58 +02:00
Bastian Kleineidam
db7ea6872a Refactor internal URL pattern matcher into function. 2011-04-27 08:34:15 +02:00
Bastian Kleineidam
719441cca5 Make module detection more robust and use it when possible. 2011-04-20 09:08:11 +02:00
Bastian Kleineidam
6a544f2d69 Only allow redirections to FTP, HTTP and HTTPS URLs. 2011-04-19 07:01:55 +02:00
Bastian Kleineidam
84f6d56a49 Print level in loggers xml, csv and sql. 2011-04-09 10:51:03 +02:00
Bastian Kleineidam
c0732e3d37 Do not print empty country information. 2011-04-06 17:22:48 +02:00
Bastian Kleineidam
82e5ba8ce6 Add warning tag attribute in XML loggers. 2011-03-15 13:42:21 +01:00
Bastian Kleineidam
f4f921384e Updated copyright 2011-03-13 07:52:18 +01:00
Bastian Kleineidam
502430489a Add url2pathname workaround for Windows. 2011-03-12 16:33:48 +01:00
Bastian Kleineidam
7b33cfac7b Use stripped URL base constructing absolute URL. 2011-03-11 15:17:36 +01:00
Bastian Kleineidam
78ea8d5594 Remove unnecessary call to url2pathname(). 2011-03-11 12:28:33 +01:00
Bastian Kleineidam
ae109ed994 Correct conversion between URL and filename paths. 2011-03-11 10:38:17 +01:00
Bastian Kleineidam
420c21c2de Strip leading and trailing whitespace from URLs. 2011-03-07 12:33:09 +01:00
Bastian Kleineidam
21e4824f65 Fix typo calling get_temp_file() function. 2011-03-07 09:57:40 +01:00
Bastian Kleineidam
de5d1757f0 Add workaround for buggy IIS HEAD support. 2011-02-24 11:12:59 +01:00
Bastian Kleineidam
c89bd05651 Remove unused variables and imports. 2011-02-19 11:46:20 +01:00
Bastian Kleineidam
2ec312301e Fix linkcheck.dns py2exe packaging. 2011-02-18 17:26:00 +01:00
Bastian Kleineidam
0d4377d1ba Support Google Chrome Bookmark files. 2011-02-15 18:26:00 +01:00
Bastian Kleineidam
25b6dc2e57 Refactor bookmark parsing code into own package. 2011-02-15 17:31:42 +01:00
Bastian Kleineidam
2dfe62afa2 Updated copyright. 2011-02-14 21:07:07 +01:00
Bastian Kleineidam
c5884b8d87 Add function documentation. 2011-02-14 21:06:34 +01:00
Bastian Kleineidam
85f3690068 Updated copyright. 2011-02-11 14:00:31 +01:00
Bastian Kleineidam
db6a3669b3 Correctly detect empty FTP paths as directories. 2011-02-11 12:35:53 +01:00
Bastian Kleineidam
362c7a1d9d Preselect filename on save dialog when editing file:// URLs. 2011-02-09 08:46:09 +01:00
Bastian Kleineidam
4a0c63aa56 Fix joining of URLs when parent URL has CGI parameter. 2011-02-08 21:25:55 +01:00
Bastian Kleineidam
71b15b70f4 Updated copyright 2011-01-06 09:59:57 +01:00
Bastian Kleineidam
5f70b7210f Add tempfile utility function. 2011-01-06 09:52:11 +01:00
Bastian Kleineidam
d011d1524c Parse PHP files recursively. 2010-12-28 17:11:29 +01:00
Bastian Kleineidam
fd3fe8dcaa Fix missing content types for cached URLs. 2010-12-23 07:37:36 +01:00
Bastian Kleineidam
2a4b60de4d Remove unused imports. 2010-12-22 13:06:24 +01:00
Bastian Kleineidam
84e4e3b28a Fix regression from last commit in this file. 2010-12-22 13:06:10 +01:00
Bastian Kleineidam
0d8a583e39 Fix internal pattern for file URLs (regression from commit 90e0f4e) 2010-12-21 21:10:31 +01:00
Bastian Kleineidam
6090e1a66c Print anchor in __str__() 2010-12-21 20:55:49 +01:00
Bastian Kleineidam
1ebd4d1fc4 Simplify code. 2010-12-21 20:55:35 +01:00
Bastian Kleineidam
90e0f4e5cc Detect filenames with spaces as internal links. 2010-12-21 07:05:12 +01:00
Bastian Kleineidam
9ea35241c0 Set correct scheme on file links. 2010-12-21 01:23:50 +01:00
Bastian Kleineidam
128f8eb6e4 Move firefox routines to firefox module. 2010-12-21 00:02:12 +01:00
Bastian Kleineidam
7c08290c44 Fix broken anchor checking. 2010-12-20 19:55:26 +01:00
Bastian Kleineidam
0b8f8d52b2 Check for empty URL before determining content type. 2010-12-18 08:26:59 +01:00
Bastian Kleineidam
224061e284 Fix to_wire by looking of URL parts have been initialized. 2010-12-15 13:24:12 +01:00
Bastian Kleineidam
7c55351511 Add get_content_type methods to subclasses. 2010-12-15 07:54:44 +01:00
Bastian Kleineidam
2b2121b9ed Added content type and domain to URL logging info. 2010-12-14 20:30:53 +01:00
Bastian Kleineidam
01184784ef Remove warning about Unicode domains which are more widely supported now. 2010-12-11 07:58:15 +01:00
Bastian Kleineidam
9e88377584 Remove stray raise statement from previous commit. 2010-11-26 21:35:49 +01:00
Bastian Kleineidam
c5676f0297 Catch socket errors when closing SMTP connections. 2010-11-26 19:51:26 +01:00
Bastian Kleineidam
5c9c15071a Limit FTP download file size. 2010-11-25 20:44:41 +01:00
Bastian Kleineidam
0cf22e5242 Limit FTP download file size. 2010-11-25 20:44:14 +01:00
Bastian Kleineidam
6fac69cddb Fall back to GET when connection is reset. 2010-11-21 19:50:51 +01:00
Bastian Kleineidam
03034ddc1c Updated copyright 2010-11-21 11:25:07 +01:00
Bastian Kleineidam
04f9c1b854 Use urlparse.parse_qs() instead of cgi.parse_qs() 2010-11-21 10:43:47 +01:00
Bastian Kleineidam
147bf31e1e Check for allowed HTTP GET method before parsing anchors in HTML file contents. 2010-11-17 19:13:26 +01:00
Bastian Kleineidam
17ce930611 Ignore irc:// URLs. 2010-11-10 19:56:31 +01:00
Bastian Kleineidam
2fde5bea8c Updated copyright 2010-11-06 18:02:56 +01:00
Bastian Kleineidam
4f5c957e43 Fix check of external domain after HTTP redirect. 2010-11-06 18:00:49 +01:00
Bastian Kleineidam
57ffa6bf97 Allow both redirection www.example.com -> example.com and vice versa. 2010-11-06 17:55:49 +01:00
Bastian Kleineidam
280b7892ef Remove unused NNTP warning. 2010-11-06 17:39:22 +01:00
Bastian Kleineidam
1188e0be2e Retry NNTP connections on temporary errors. 2010-11-06 17:26:40 +01:00
Bastian Kleineidam
23b20306e9 Remove duplicate HTTP response codes. 2010-11-01 09:27:53 +01:00
Bastian Kleineidam
c5f93a561d Fix debug message formatting. 2010-11-01 05:59:04 +01:00
Bastian Kleineidam
f14340a0a8 Do not check content of already cached URLs. 2010-10-27 19:52:48 +02:00
Bastian Kleineidam
1f81124dfa Fix typo. 2010-10-27 19:23:14 +02:00
Bastian Kleineidam
23403f09bb Do not print warning for HTTP to HTTPS or HTTPS to HTTP redirects. 2010-10-27 14:44:05 +02:00
Bastian Kleineidam
b2cf40151f Improved redirection warning text. 2010-10-27 09:15:46 +02:00
Bastian Kleineidam
d9e981e497 Don't log a warning if commandline URL has been redirected. 2010-10-26 16:24:27 +02:00
Bastian Kleineidam
4375d35328 Add warning about unsupported HTTP authentication, and revert the realm changes. 2010-10-25 22:41:31 +02:00
Bastian Kleineidam
332fa4f8f9 Prepare multi-realm auth configuration. 2010-10-25 22:07:16 +02:00
Bastian Kleineidam
2a7292845c Improved info message about sent cookies; do not report the retrieved cookie information. 2010-10-13 22:32:50 +02:00
Bastian Kleineidam
a8aa3bdb00 Another fix to ensure get_content() is only called when allowed. 2010-10-13 22:14:43 +02:00
Bastian Kleineidam
61e611e4bf Prevent unallowed content read when checking for robots.txt allowance in HTML files. 2010-10-12 00:40:34 +02:00
Bastian Kleineidam
1d0db02192 Refactor getting user and password for an URL. 2010-10-11 20:11:15 +02:00
Bastian Kleineidam
e494d6bbb6 Move MIME type detection into fileutil.py module, and use mimetools for detection. 2010-10-03 08:47:48 +02:00
Bastian Kleineidam
e0f4097eb0 Ensure HttpUrl.set_title_from_content() is only called when the content is allowed to be retrieved. 2010-09-29 19:26:03 +02:00
Bastian Kleineidam
840538d12a Remove uneeded check for HTML content. 2010-09-29 19:25:14 +02:00
Bastian Kleineidam
279a1eae70 Only add geoip info for non-empty hostnames. 2010-09-29 15:59:57 +02:00
Bastian Kleineidam
cc848cdb33 Fix import for moved geoip module. 2010-09-29 15:17:27 +02:00
Bastian Kleineidam
ffcd274087 Updated copyright 2010-09-05 21:02:51 +02:00
Bastian Kleineidam
8a1ac26c85 Warn about obfuscated IP numbers. 2010-09-05 20:11:02 +02:00
Bastian Kleineidam
5284017d67 Only fallback to HTTP GET when robots.txt sallows it. 2010-09-04 18:09:59 +02:00
Bastian Kleineidam
8a074aeea9 Work around Python 2.6+ urljoin bug. 2010-08-31 09:16:24 +02:00
Bastian Kleineidam
c3b8ff00b3 Check content and recursion in one try/except to avoid multiple errors when getting page content. 2010-08-31 06:52:08 +02:00
Bastian Kleineidam
60f7af4598 Allow redirections to external URLs with same domain. 2010-08-13 01:22:18 +02:00
Bastian Kleineidam
1faedafb33 Fix data size for HTTP requests. 2010-08-04 00:06:25 +02:00
Bastian Kleineidam
c086f49cea Catch KeyError when quoting URLs of index.html. 2010-07-30 20:12:52 +02:00
Bastian Kleineidam
4678802a81 Do not truncate UNC filepaths 2010-07-30 20:07:11 +02:00
Bastian Kleineidam
761b292e37 Added skype: to list of recognized but ignored URL schemes. 2010-07-29 20:26:04 +02:00
Bastian Kleineidam
0f92b76290 Remove the unnormed URL warning. 2010-07-29 20:20:59 +02:00
Bastian Kleineidam
7ad4f7c220 Compare size from meta info and content data. 2010-07-29 19:53:41 +02:00
Bastian Kleineidam
8413b427e9 Rename some warnings, and add size unequality warning. 2010-07-29 19:53:15 +02:00
Bastian Kleineidam
7536472797 Send correct host header when using http proxy. 2010-07-29 06:50:35 +02:00
Bastian Kleineidam
41e2e1a448 Add new warning to warning list. 2010-07-28 13:47:58 +02:00
Bastian Kleineidam
d9bfd25a68 Add warning if content size is zero 2010-07-28 08:19:55 +02:00
Bluebird75
28f4514b67 Use object with __slots__ for wire-format of UrlBase objects. 
Saves memory since UrlBase wire-format objects are used for
logging and thus often created.

Signed-off-by: Bastian Kleineidam <calvin@debian.org>
 2010-03-27 00:07:19 +01:00
Bastian Kleineidam
3370ea1562 Reflect changes in httplib2.py: use buffered read in httplib response object and use bad status line exception attribute. 2010-03-26 20:50:38 +01:00
Bastian Kleineidam
c4c098bd83 pep8-ify the source a little more 2010-03-13 08:47:12 +01:00
Bastian Kleineidam
37b4e97012 Revert "Only parse anchors if both --anchors option is given and the current link has an anchor." 
This reverts commit b238527d54.
 2010-03-10 00:04:02 +01:00
Bastian Kleineidam
b238527d54 Only parse anchors if both --anchors option is given and the current link has an anchor. 2010-03-09 11:45:50 +01:00
Bastian Kleineidam
57397e938b Improved linkname parsing by adding a new peek() HTML parser function. 2010-03-09 11:31:12 +01:00
Bastian Kleineidam
074b5ded32 Support UTF-8 encoded filenames in FTP servers. 2010-03-09 08:15:29 +01:00
Bastian Kleineidam
c88791b815 Fix support for non-standard FTP ports. 2010-03-09 07:49:05 +01:00
Bastian Kleineidam
51a0ef0ad4 Speed up HTML parsing by stopping early and adding callbacks. 2010-03-08 09:04:33 +01:00
Bastian Kleineidam
b8b0398dd2 Ensure redirected URL is Unicode encoded. 2010-03-07 22:11:55 +01:00
Bastian Kleineidam
c8e6995ecd Support HTTPS proxies. 2010-03-07 21:06:10 +01:00
Bastian Kleineidam
1e15e55689 Fix errors in Word file parsing. 2010-03-07 19:43:08 +01:00
Bastian Kleineidam
6a2fcf8ae9 Parse links in Word files. 2010-03-07 19:20:51 +01:00
Bastian Kleineidam
34a2f4a15d Disable and deprecated the --no-proxy-for option. 2010-03-07 17:45:48 +01:00
Bastian Kleineidam
796cf0a7cd Updated copyright year 2010-03-07 11:59:18 +01:00
Bastian Kleineidam
af6cb287d7 Only warn about missing emails in mailto: URLs. 2010-03-07 10:43:29 +01:00
Bastian Kleineidam
3d5c114f14 Warn on permament redirections even when URL is outside of domain filter. 2010-03-07 09:36:21 +01:00
Bastian Kleineidam
2d73b907f1 Retry HTTP when server sent empty status line; should fix most of the BadStatusLine errors that are sporadically encountered. 2010-03-06 10:23:34 +01:00
Bastian Kleineidam
77daf80e82 Add url encoding parameter 2009-11-28 11:56:35 +01:00
Bastian Kleineidam
5e06b6b8d4 Updated FSF address in GPL blurb 2009-07-24 23:58:20 +02:00
Bastian Kleineidam
e6f43b6822 Fixed the no_proxy handling and added changelog entry 2009-07-24 07:19:49 +02:00
Bastian Kleineidam
7f67027abf ignore the fragment part (ie. the anchor) of URIs when 
+  getting and caching content
 2009-06-26 07:22:36 +02:00
Bastian Kleineidam
c7b7af877f Read Mozilla bookmark titles correctly from places.sqlite. 2009-05-20 07:50:46 +02:00
Bastian Kleineidam
59ffbd43f0 Use AttrDict for transport object in loggers. 2009-03-07 09:43:55 +01:00
Bastian Kleineidam
7a59763508 Remove unused SetList container 2009-03-07 00:42:27 +01:00
Bastian Kleineidam
2351506752 Use plain list for info strings. 2009-03-07 00:19:19 +01:00
Bastian Kleineidam
897b68ae9b Fix copying of httpurl info 2009-03-07 00:17:17 +01:00
Bastian Kleineidam
88dbcb30cd Remove unused url_data.info tags - the tags were always None 2009-03-06 21:20:09 +01:00
Bastian Kleineidam
0b5f525f76 Print NNTP server welcome string as info 2009-03-06 20:57:35 +01:00
Bastian Kleineidam
4ee0fb0181 Add NNTP debugging. 2009-03-06 20:53:12 +01:00
Bastian Kleineidam
0bc2fbb47a Only try 3 times connecting to a busy NNTP server, not 5 times. 2009-03-06 20:52:53 +01:00
Bastian Kleineidam
29adfe92fd Minor syntax fix 2009-03-06 20:14:50 +01:00
Bastian Kleineidam
6024f2e43e Add missing reset of self.reused_connection flag 2009-03-06 20:10:03 +01:00
Bastian Kleineidam
ba160350dd Introduced transport object API for logging. 2009-03-06 19:30:58 +01:00
Bastian Kleineidam
58925b21d3 Improved persistent connection handling by retrying closed connections. 2009-03-06 08:15:34 +01:00
Bastian Kleineidam
29599e4c74 Make sure persistent connection will not close after reading contents. 2009-03-05 19:15:44 +01:00
Bastian Kleineidam
bf9ed8c659 Make sure file descriptors are closed after decoding HTTP content. 2009-03-05 19:15:03 +01:00
Bastian Kleineidam
b8944e493a Use new exception log keyword when logging errors 2009-03-02 13:18:36 +01:00
Bastian Kleineidam
a9335fb3e8 Make file list an iterator, and add missing slash if needed to manually given file URLs. 2009-03-02 08:02:27 +01:00
Bastian Kleineidam
7862147ca3 Fix showing content size. 2009-03-01 23:04:48 +01:00
Bastian Kleineidam
8caa601a7e Python 3.0 compatibility: use exc.args[] instead of exc[] 2009-02-24 12:41:45 +01:00
Bastian Kleineidam
2c9b8d6858 Use slash as path separator in file names 2009-02-24 12:41:28 +01:00
Bastian Kleineidam
323958951c Add name to unnamed file URLs. 2009-02-20 14:03:34 +01:00
calvin
2e918a7b7a Added email syntax check. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3960 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-02-18 15:35:23 +00:00
calvin
7214943f38 Remove wrong function return type documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3959 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-02-18 15:34:46 +00:00
calvin
7e5a2ea23b Remove unused file 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3930 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-24 17:35:06 +00:00
calvin
e03df9e709 Removed gopher URL checking. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3929 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-24 17:34:18 +00:00
calvin
c6cb09c4aa Add missing import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3900 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 19:41:42 +00:00
calvin
1c50cf288a Ignore DNS MX lookup failures in py2exe. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3899 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 18:50:14 +00:00
calvin
cc25deac12 Only accept MX dns response types when asking for MX servers. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3895 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 17:53:10 +00:00
calvin
979132c9b5 Catch all DNS exceptions when resolving MX hosts. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3894 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 15:13:55 +00:00
calvin
a26ca4c23a Replace C ftpparse module with Python implementation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3892 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 14:11:17 +00:00
calvin
e9805dbd8a Updated copyright year to 2009 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3887 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-08 14:18:03 +00:00
calvin
8d5d4827c3 Change ftpparse import to avoid py2exe load error. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3883 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-08 12:28:39 +00:00
calvin
209d5abc18 fix timeouts by testing earlier for persistent connections with HEAD 
HEAD requests never have a body; nevertheless the http lib tries to
read() from them. This times out on some servers of course. Fix is
not to let those connections be persistent.

git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3871 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-29 08:14:28 +00:00
calvin
c20e706761 Made some format changes on translated strings. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3870 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-28 20:22:48 +00:00
calvin
1abc2c504d Filter invalid mozilla bookmark urls from places.sqlite 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3869 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-28 10:54:16 +00:00
calvin
c3b6fc5aa4 Readd 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3867 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 21:30:10 +00:00
calvin
42c3e71329 Improved and tested opera bookmark parser 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3863 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 07:52:02 +00:00
calvin
9ab895751f Support parsing of Firefox 3 bookmark files 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3862 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 07:51:22 +00:00
calvin
97cf700e04 Fixed wrong cookie debugging format line. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3849 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-07-13 12:51:56 +00:00
calvin
523ee87f0c Add missing return in is_absolute_path() 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3846 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-07-09 19:17:33 +00:00
calvin
f68872f559 Improved detection of absolute Windows paths. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3844 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-07-09 19:00:02 +00:00
calvin
84355f7b94 Catch original httplib errors too since it is used indirectly by urllib functions. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3833 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-30 23:17:38 +00:00
calvin
b30fb3b09c Remove duplicate code in http checker. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3820 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 19:52:09 +00:00
calvin
caf8ba6297 Really allow parsing of XHTML files; I forgot some places to adjust the MIME checking. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3818 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 13:03:48 +00:00
calvin
a6deeeb8a5 Support parsing of HTML pages served with content type application/xhtml+xml 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3817 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 09:39:49 +00:00
calvin
ff41aa8d9f Lower the MIME content-type info from HTTP headers befure using it 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3816 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 09:38:09 +00:00
calvin
d26386d03f Catch errors when getting content for title. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3814 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-12 15:38:26 +00:00
calvin
a880939c40 Initialize variables in reset(), not in subsequent methods 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3796 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-08 09:27:13 +00:00
calvin
290528b84f Added title attribute to URL data. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3790 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-07 13:07:56 +00:00
calvin
99269d12cc Add base method for Url.get_title() 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3788 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-07 13:07:38 +00:00
calvin
5f4d61e018 Use keyword arguments in translation strings. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3780 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:44:40 +00:00
calvin
97772c9700 Improved email check messages. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3779 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:44:07 +00:00
calvin
2e4d0894fc Stop checking a list of emails at the first invalid one. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3778 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:43:20 +00:00
calvin
e6e51dbc6b Overwrite old results when checking a list of emails. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3777 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:42:52 +00:00
calvin
66ff422f6b Allow overwriting of an old check result. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3776 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:42:38 +00:00
calvin
7297519b04 Remove or replace unused variables. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3772 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-22 12:10:08 +00:00
calvin
9352dbf5e4 Move test files to separate module 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3763 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-20 17:01:16 +00:00
calvin
dbb498a395 Add virus checking 
New option --scan-virus to check the content of URLs for
viruses with ClamAV.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3753 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-20 08:57:37 +00:00
calvin
bacb59597e Use relative imports from Python 2.5 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3750 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-09 06:16:03 +00:00
calvin
b96e8120d6 Add W3C Validator checks 
Add new options --check-html-w3 and --check-css-w3 to allow checking
of HTML and CSS pages with the online W3C validators.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3748 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-08 10:36:54 +00:00