Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-16 18:41:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
5618 commits 1 branch 0 tags 24 MiB
Commit graph

771 commits

Author SHA1 Message Date
Bastian Kleineidam
6024f2e43e Add missing reset of self.reused_connection flag 2009-03-06 20:10:03 +01:00
Bastian Kleineidam
ba160350dd Introduced transport object API for logging. 2009-03-06 19:30:58 +01:00
Bastian Kleineidam
58925b21d3 Improved persistent connection handling by retrying closed connections. 2009-03-06 08:15:34 +01:00
Bastian Kleineidam
29599e4c74 Make sure persistent connection will not close after reading contents. 2009-03-05 19:15:44 +01:00
Bastian Kleineidam
bf9ed8c659 Make sure file descriptors are closed after decoding HTTP content. 2009-03-05 19:15:03 +01:00
Bastian Kleineidam
b8944e493a Use new exception log keyword when logging errors 2009-03-02 13:18:36 +01:00
Bastian Kleineidam
a9335fb3e8 Make file list an iterator, and add missing slash if needed to manually given file URLs. 2009-03-02 08:02:27 +01:00
Bastian Kleineidam
7862147ca3 Fix showing content size. 2009-03-01 23:04:48 +01:00
Bastian Kleineidam
8caa601a7e Python 3.0 compatibility: use exc.args[] instead of exc[] 2009-02-24 12:41:45 +01:00
Bastian Kleineidam
2c9b8d6858 Use slash as path separator in file names 2009-02-24 12:41:28 +01:00
Bastian Kleineidam
323958951c Add name to unnamed file URLs. 2009-02-20 14:03:34 +01:00
calvin
2e918a7b7a Added email syntax check. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3960 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-02-18 15:35:23 +00:00
calvin
7214943f38 Remove wrong function return type documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3959 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-02-18 15:34:46 +00:00
calvin
7e5a2ea23b Remove unused file 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3930 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-24 17:35:06 +00:00
calvin
e03df9e709 Removed gopher URL checking. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3929 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-24 17:34:18 +00:00
calvin
c6cb09c4aa Add missing import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3900 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 19:41:42 +00:00
calvin
1c50cf288a Ignore DNS MX lookup failures in py2exe. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3899 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 18:50:14 +00:00
calvin
cc25deac12 Only accept MX dns response types when asking for MX servers. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3895 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 17:53:10 +00:00
calvin
979132c9b5 Catch all DNS exceptions when resolving MX hosts. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3894 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 15:13:55 +00:00
calvin
a26ca4c23a Replace C ftpparse module with Python implementation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3892 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-10 14:11:17 +00:00
calvin
e9805dbd8a Updated copyright year to 2009 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3887 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-08 14:18:03 +00:00
calvin
8d5d4827c3 Change ftpparse import to avoid py2exe load error. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3883 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-08 12:28:39 +00:00
calvin
209d5abc18 fix timeouts by testing earlier for persistent connections with HEAD 
HEAD requests never have a body; nevertheless the http lib tries to
read() from them. This times out on some servers of course. Fix is
not to let those connections be persistent.

git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3871 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-29 08:14:28 +00:00
calvin
c20e706761 Made some format changes on translated strings. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3870 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-28 20:22:48 +00:00
calvin
1abc2c504d Filter invalid mozilla bookmark urls from places.sqlite 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3869 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-28 10:54:16 +00:00
calvin
c3b6fc5aa4 Readd 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3867 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 21:30:10 +00:00
calvin
42c3e71329 Improved and tested opera bookmark parser 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3863 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 07:52:02 +00:00
calvin
9ab895751f Support parsing of Firefox 3 bookmark files 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3862 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 07:51:22 +00:00
calvin
97cf700e04 Fixed wrong cookie debugging format line. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3849 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-07-13 12:51:56 +00:00
calvin
523ee87f0c Add missing return in is_absolute_path() 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3846 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-07-09 19:17:33 +00:00
calvin
f68872f559 Improved detection of absolute Windows paths. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3844 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-07-09 19:00:02 +00:00
calvin
84355f7b94 Catch original httplib errors too since it is used indirectly by urllib functions. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3833 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-30 23:17:38 +00:00
calvin
b30fb3b09c Remove duplicate code in http checker. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3820 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 19:52:09 +00:00
calvin
caf8ba6297 Really allow parsing of XHTML files; I forgot some places to adjust the MIME checking. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3818 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 13:03:48 +00:00
calvin
a6deeeb8a5 Support parsing of HTML pages served with content type application/xhtml+xml 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3817 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 09:39:49 +00:00
calvin
ff41aa8d9f Lower the MIME content-type info from HTTP headers befure using it 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3816 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-16 09:38:09 +00:00
calvin
d26386d03f Catch errors when getting content for title. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3814 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-12 15:38:26 +00:00
calvin
a880939c40 Initialize variables in reset(), not in subsequent methods 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3796 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-08 09:27:13 +00:00
calvin
290528b84f Added title attribute to URL data. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3790 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-07 13:07:56 +00:00
calvin
99269d12cc Add base method for Url.get_title() 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3788 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-06-07 13:07:38 +00:00
calvin
5f4d61e018 Use keyword arguments in translation strings. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3780 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:44:40 +00:00
calvin
97772c9700 Improved email check messages. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3779 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:44:07 +00:00
calvin
2e4d0894fc Stop checking a list of emails at the first invalid one. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3778 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:43:20 +00:00
calvin
e6e51dbc6b Overwrite old results when checking a list of emails. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3777 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:42:52 +00:00
calvin
66ff422f6b Allow overwriting of an old check result. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3776 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-27 19:42:38 +00:00
calvin
7297519b04 Remove or replace unused variables. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3772 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-22 12:10:08 +00:00
calvin
9352dbf5e4 Move test files to separate module 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3763 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-20 17:01:16 +00:00
calvin
dbb498a395 Add virus checking 
New option --scan-virus to check the content of URLs for
viruses with ClamAV.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3753 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-20 08:57:37 +00:00
calvin
bacb59597e Use relative imports from Python 2.5 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3750 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-09 06:16:03 +00:00
calvin
b96e8120d6 Add W3C Validator checks 
Add new options --check-html-w3 and --check-css-w3 to allow checking
of HTML and CSS pages with the online W3C validators.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3748 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-08 10:36:54 +00:00
calvin
df9f31dcb1 Only check HTML/CSS syntax of intern URLs 
The HTML and CSS syntax check now only applies to URLs
which match those given on the command line.
This makes checking of personal pages easier.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3743 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-29 17:48:47 +00:00
calvin
ac4d09f83d Fix errors in CSS and HTML syntax check 
Properly encode the warning messages as Unicode, and prevent
overwriting of the "log" module with a local variable.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3742 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-29 17:48:22 +00:00
calvin
92c74ece4d Send HTTP Referer header to both http and https URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3741 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-29 13:33:35 +00:00
calvin
5d8bdaaa1f Use generators instead of lists where possible 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3739 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-28 00:26:02 +00:00
calvin
3eac1be9ab Require and use Python 2.5 
Use Python 2.5 features and get rid of old compat code. Also some
code cleanups have been made.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3737 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-27 11:39:21 +00:00
calvin
72db31e546 Only check syntax of valid URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3726 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-25 07:53:11 +00:00
calvin
973da91f44 Source code cleanup: use or remove unused variables 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3724 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-25 07:49:52 +00:00
calvin
e266a65b64 Fix css check 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3723 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-24 10:46:19 +00:00
calvin
62efec3b35 Added CSS syntax check. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3719 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-24 09:44:18 +00:00
calvin
cce6affa17 Add --check-html option to check the HTML syntax. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3718 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-23 23:04:31 +00:00
calvin
df857aab8d Intern patterns now accept URLs with and without "www." prefixes 
as default. This allows sites to check that use both variants.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3714 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-21 09:18:58 +00:00
calvin
5a2f89fa3d Add redirect warning for commandline URLs 
If URLs given on the commandline are redirected, the automatic
intern patterns might not match anymore. A warning makes this
more prominent.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3712 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-21 09:18:36 +00:00
calvin
8ae6d94b45 Improved error messages for exceptions 
Prepend the exception name before the error message of exceptions.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3694 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-19 07:47:00 +00:00
calvin
4968f1b3cd Prevent empty exception values. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3690 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-18 07:42:24 +00:00
calvin
ba148a9d71 Proper MX DNS request fallback 
Properly fall back to DNS A requests when no MX host could be found
for a mailto: URL.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3689 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-14 22:34:46 +00:00
calvin
9b7cf763ff Fix test for new www.example.org URL 
Fix test data using www.example.org instead of imadoofus.org URLs.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3688 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-14 22:34:29 +00:00
calvin
4055721fd4 Use internal gzip2 module 
Use the internal gzip replacement module gzip2 for all GzipFile handling.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3685 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-14 22:33:55 +00:00
calvin
1f5a2d47ea Syntax cleanups 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3682 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-03-25 21:24:21 +00:00
calvin
e178405748 Use example.{com,org} for example URLs 
Use the guaranteed not available example.com and example.org DNS names
in example URLs.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3681 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-03-25 21:23:34 +00:00
calvin
4ce0ddd166 Changes for future Python 3.x compatibility 
Replace backticks with repr(), replace .has_key() with "in".


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3680 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-03-19 10:22:57 +00:00
calvin
91a0aad5d8 Fix buggy persistent HTTP connections 
Workaround for buggy servers that break protocol synchronization of
persistent HTTP connections.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3677 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-03-14 13:28:43 +00:00
calvin
1730097265 Prevent Unicode errors for non-ASCII emails 
Prevent Unicode errors when email address contains non-ASCII characters.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3673 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-03-02 23:38:02 +00:00
calvin
860def8d34 Remove superfluous path slash 
Really fix the test_misc unit test by removing a superluous path slash.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3672 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-02-08 10:38:46 +00:00
calvin
294261d80a Replace hardcoded test paths for test_misc 
Replace the hardcoded test paths with variables. Fixes failures
in the test_misc unit test.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3670 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-02-08 10:38:11 +00:00
calvin
7cf9723b10 don't parse <script for=''> as URL 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3659 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-01-11 16:45:30 +00:00
calvin
6499cb1a63 updated copyright year 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3658 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-01-02 14:31:19 +00:00
calvin
c99b9b1e8f added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3657 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-28 08:25:40 +00:00
calvin
c971ebdabf Added Shockwave Flash (SWF) parsing 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3656 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-28 02:12:48 +00:00
calvin
30d2b4f520 HTTP content data is only considered valid for parsing if the request was not redirected and is a GET request. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3633 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-13 10:50:13 +00:00
calvin
41bc0b2b32 use 'self.data is None' to test if data is already read or not 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3631 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-03 14:09:20 +00:00
calvin
5591bbe052 fix self.downloadtime to self.dltime 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3630 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-03 14:01:36 +00:00
calvin
8e6c6455ab add missing import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3626 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-01 15:56:15 +00:00
calvin
09ce26d5fe removed debug flag, test the LOG_CHECK logger for debug settings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3623 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-12-01 15:50:59 +00:00
calvin
8d2dc781e1 Ensure unused or expired connections are closed. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3617 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-30 16:42:41 +00:00
calvin
f8a54faae9 make sure internpat does not remove a trailing slash, which results in checking of URLs that are not a prefix of the given URL. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3613 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-30 10:37:14 +00:00
calvin
9cf3314eab Use constants for warning tags, avoiding typos in string constants. And move the constants into a separate module const.py 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3611 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-29 07:50:22 +00:00
calvin
e007ea5dae fix warning typo 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3610 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-29 06:37:40 +00:00
calvin
fcde8bd4d6 try to detect unknown URL schemes instead of manually setting the assume_local flag 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3609 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-28 18:46:50 +00:00
calvin
a50784042f make sure URL to test for IDNA encoding errors has non-ascii characters 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3608 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-28 18:40:33 +00:00
calvin
6a0960aa66 only store parser contents in LinkFinder handler, not in all handlers 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3602 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-14 19:06:39 +00:00
calvin
a1d911127b remove comments from CSS files before parsing for links 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3601 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-14 18:46:14 +00:00
calvin
cb588a3c5d replace tabs with spaces 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3598 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-14 17:55:55 +00:00
calvin
ce8b963dd9 more code cleanups and documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3596 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-14 17:46:18 +00:00
calvin
370749cafb cleanup the code and add some documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3595 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-14 17:34:50 +00:00
calvin
e9c973fe06 Honor urllib.proxy_bypass() when ignoring proxy settings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3583 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-10-23 22:56:44 +00:00
calvin
2edfaea03e Read complete body data on persistent connections, else subsequent requests could fail. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3568 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-08-08 19:33:10 +00:00
calvin
2b94c0c161 Assume missing HEAD requests for Zope server on text/plain content type 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3567 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-08-08 18:26:55 +00:00
calvin
5aed37dada use german server for faster testing (at least for me) 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3553 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-06-13 00:35:32 +00:00
calvin
df48d4a905 bump up copyright year 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3534 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-01-01 14:57:38 +00:00
calvin
c217b6d441 don't set result on self.get_content() redirections 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3515 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-11-17 20:42:00 +00:00
calvin
698f7183bc fix vrfy error message 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3507 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-10-19 19:57:29 +00:00
calvin
bef2494211 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3482 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-24 10:13:59 +00:00
calvin
1883b79303 follow redirections when getting HTTP contents 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3473 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-21 09:27:38 +00:00
calvin
5ad59225a0 use dictionaries for translations with multiple arguments 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3460 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-16 09:12:59 +00:00
calvin
576d404ce2 close non-idle connections 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3453 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 21:27:19 +00:00
calvin
04f89d0668 use get_url_from helper alias 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3451 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 21:18:49 +00:00
calvin
72d198efcb don't send keep-alive header, it breaks some tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3450 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 21:18:29 +00:00
calvin
86514bb882 activate asset 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3448 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 20:21:06 +00:00
calvin
ba7eaeae09 ignore geo location info lines in test output 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3445 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 20:03:54 +00:00
calvin
6348205dcc add persistent connections back to the connection cache, close all others 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3444 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 19:59:30 +00:00
calvin
d6676ab0a0 more response closing, and cleanups 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3443 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 19:51:02 +00:00
calvin
6fe2db6755 use unicode_safe alias helper 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3442 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 19:46:55 +00:00
calvin
27a8869783 use helper alias for unicode_safe 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3441 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 19:34:03 +00:00
calvin
15dfaf35cb cleanup 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3438 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 16:36:44 +00:00
calvin
4b818cb4b3 Detect more cases to close the connection, and close response objects 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3437 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 16:35:53 +00:00
calvin
da15b15923 Split off the host wait time function, and use it with a separate lock 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3434 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 12:18:24 +00:00
calvin
0a5c03536d remove unneeded logger arguments 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3425 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-08-21 20:50:10 +00:00
calvin
f78d9bb337 s/fields/parts/ for logger arguments, and supporess the last modified info 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3424 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-08-21 20:50:00 +00:00
calvin
adc4e8c0e8 quote base reference URL, with tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3402 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-07-18 08:07:46 +00:00
calvin
c6f01faab5 improved debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3401 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-07-18 07:43:36 +00:00
calvin
7a31cb7ede add tests for file output 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3351 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-06-05 19:47:09 +00:00
calvin
7781fe88ce use relative imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3350 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-06-05 19:46:47 +00:00
calvin
7667f3402f send short keep-alive header value for test server 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3349 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-06-05 19:46:23 +00:00
calvin
d95d8c3d96 correctly handle internal errors 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3338 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-06-03 01:14:05 +00:00
calvin
8763a42063 skip a test on nt platforms 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3327 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-29 16:08:37 +00:00
calvin
850684b1e0 datadir as url path 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3325 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-29 14:00:45 +00:00
calvin
2888c34859 split file tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3324 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-29 13:54:40 +00:00
calvin
e211d3fd6c fix internal error call 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3314 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-25 11:33:23 +00:00
calvin
98597c267d quote result line 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3272 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-19 17:21:20 +00:00
calvin
3142663135 added tests for UnicodeError 'label too long' 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3270 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-19 17:13:28 +00:00
calvin
7e1e01bd36 do not catch UnicodeError, handle that intern 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3269 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-19 17:13:16 +00:00
calvin
2c13d7cac1 norm test urls 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3267 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:43:55 +00:00
calvin
608f8ba1c3 prepare filenames as URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3266 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:38:57 +00:00
calvin
37615dba02 use datadir, curdir placeholders 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3265 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:35:52 +00:00
calvin
14a29fb015 prepare filenames as URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3264 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:35:36 +00:00
calvin
0ba1520d13 fix filename for test 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3263 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:14:30 +00:00
calvin
1dbc97abe7 script moving 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3262 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:10:16 +00:00
calvin
23879d78d4 adjust test result for new cache optimization 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3259 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 19:14:54 +00:00
calvin
cd8886c77f adjust test results for optimized cache 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3258 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 18:55:03 +00:00
calvin
2ec5c054fe merge ignoredurl and errorurl into unknownurl, updated tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3237 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 19:08:40 +00:00
calvin
4ec74f6f5c added robots.txt tests for the internal HTTP server 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3232 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:47:48 +00:00
calvin
3adaf48b3d add callback for crawldelay 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3227 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:35:58 +00:00
calvin
811f5492c4 fix --pause to delay requests to the same host 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:56:13 +00:00
calvin
ad28599e57 Note if URL is missing (instead of saying it is empty) 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3220 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:15:34 +00:00
calvin
75e88c062a added --cookiefile option to set initial cookie values 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 20:56:34 +00:00
calvin
b6ad3084aa added more anchor tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3201 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:22:56 +00:00
calvin
523e6e8e43 use variables in result lines 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3174 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:49:58 +00:00
calvin
91ff370ed7 on redirection to different URL scheme take caching into account; adjust tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3173 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:47:50 +00:00
calvin
2a336f8dad put redirects in url queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3172 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:01:41 +00:00
calvin
9a431fde40 fix imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3170 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 10:14:07 +00:00
calvin
5ba1324a73 an intern pattern matches both http and https URLs now 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3169 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 10:06:33 +00:00
calvin
dc9f04e6dc adjust debug asserts 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3159 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 21:03:21 +00:00
calvin
7afe60c841 removed 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3149 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:08:35 +00:00
calvin
98a6d1eb00 added missing import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3148 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:08:06 +00:00
calvin
f002c5f965 Replace the old threading algorithm with a new one based on Queue.Queue and consumer threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3146 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:44:52 +00:00
calvin
c33e6d5907 split up invalid* tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3144 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:41:10 +00:00
calvin
d547b0ee31 fix mail tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3134 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-03 17:00:08 +00:00
calvin
4b10271233 use unittest.TestCase for standard test 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3129 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-04-24 20:16:57 +00:00
calvin
a764b1484b import cleanup 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3126 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-04-24 20:12:40 +00:00
calvin
e42f1a0cec replace sf.net with sourceforge.net 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3094 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-04-03 16:35:39 +00:00
calvin
276437c7d8 syntax cleanup 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3067 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-02-09 07:57:22 +00:00
calvin
b6c947442a documentation fixes 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3033 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-02-01 19:48:02 +00:00
calvin
a2ed1b323f do not lowercase the absolute url, caused the URL matching not to work for URLs with uppercase characters 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3022 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-02-01 18:49:31 +00:00
calvin
48cdfd32ee fix tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3011 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-01-04 01:20:20 +00:00
calvin
e92aee054c updated copyright 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3010 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-01-03 19:12:47 +00:00
calvin
388475cbe2 new-style exceptions 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2999 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-20 20:17:33 +00:00
calvin
c84a33c7ce syntax fix 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2988 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-18 09:13:23 +00:00
calvin
856ff8ef2a assert debugs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2987 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-18 08:55:42 +00:00
calvin
1cec04018f cleanup unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2985 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-18 08:40:22 +00:00
calvin
19c0a3c2ed use new cookie parsing 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2983 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-18 08:19:11 +00:00
calvin
d7a4a08659 updated tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2974 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-17 19:22:44 +00:00
calvin
9425873830 cache aliases (from redirects) 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2964 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-08 20:20:21 +00:00
calvin
791bd87f6d use LinkCheckTest 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2956 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-08 17:08:08 +00:00
calvin
4cf6f29afe unused import removed 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2953 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-07 22:40:09 +00:00
calvin
329043bc9f use datadir 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2951 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-07 22:33:14 +00:00
calvin
ddb09ab76a added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2949 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-07 21:55:16 +00:00
calvin
c86c0870d6 debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2946 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-06 22:32:31 +00:00
calvin
b733ffdb2a remove has_incoming, debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2945 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-12-06 22:32:14 +00:00
calvin
e2a454e31f use new trace module 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2939 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-11-14 22:49:37 +00:00
calvin
c049837cdd remove parser messages 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2927 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-11-04 20:05:59 +00:00
calvin
e3beb86ac1 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2911 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-17 12:18:18 +00:00
calvin
041df3a48d documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2906 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 22:54:23 +00:00
calvin
a2e422ce0d reindent 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2900 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 22:26:12 +00:00
calvin
96f4f946b2 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2898 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 21:29:17 +00:00
calvin
8ce45149fa re-add nofollow check 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2890 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 20:49:56 +00:00
calvin
93c498699d disable keyboard interrupt and SystemExit while aborting 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2888 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 20:11:36 +00:00
calvin
9e560f9ccf only return on thread wait timeout 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2887 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 20:08:14 +00:00
calvin
c585d095b4 catch SystemExit from internal errors in subthreads, and eliminate abort wrapper 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2886 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 20:06:57 +00:00
calvin
5303f94b76 ensure proxy value syntax 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2883 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 17:59:20 +00:00
calvin
50ea41b12d use preferred locale for default encoding 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2878 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 17:22:12 +00:00
calvin
753b907d51 remove unneeded imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2871 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 12:56:57 +00:00
calvin
25044ce08b print errors in the preferred encoding 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2864 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-13 12:49:52 +00:00
calvin
6dd4ac6a65 typo fixes 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2849 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-11 12:52:28 +00:00
calvin
7cf3f83c0f better recursion check debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2848 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-11 12:39:46 +00:00
calvin
978ba63999 catch UnicodeError on syntax check 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2847 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-11 12:10:32 +00:00
calvin
51f541607c catch CGI syntax errors in mailto: URLs and add a warning 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2841 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-10 23:06:07 +00:00
calvin
65150ad79e coerce to unicode after relative file handling 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2839 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-10 22:50:40 +00:00
calvin
57c9f325a9 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2836 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-10-10 22:10:10 +00:00
calvin
f01b84b894 rework the redirection routine a little, putting warnings specifically for redirections 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2786 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-20 15:51:12 +00:00
calvin
ed932cb165 debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2785 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-20 15:50:27 +00:00
calvin
317ef181f0 handle all redirections to different URL schemes, 
not just HTTP -> not HTTP, and fix a variable typo


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2780 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-20 08:53:57 +00:00
calvin
b94eec79ef use correct value for status output of checked links 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2773 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-17 23:26:13 +00:00
calvin
d347840dee use official HTTP status names 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2754 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-15 11:31:32 +00:00
calvin
359612b6fe add missing import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2751 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-14 17:53:27 +00:00
calvin
225c045815 do not cache unusual errors 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2750 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-14 17:11:13 +00:00
calvin
5d82d64c31 more result sanity check warnings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2749 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-14 13:35:18 +00:00
calvin
c140b510b1 fix warning tags 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2746 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-13 19:33:26 +00:00
calvin
f80480469f config to ignore certain warnings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2743 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-13 15:03:17 +00:00
calvin
56b2b422a9 ensure that thread name is ascii 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2718 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-05 22:06:18 +00:00
calvin
bb2b227b13 catch all errors on url check 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2717 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-05 22:05:54 +00:00
calvin
e7ca18403a work around python bug occuring with incomplete gzip streams 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2716 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-05 19:56:42 +00:00
calvin
901227d436 assign extern info to local var 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2709 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-04 22:19:52 +00:00
calvin
2e207be127 allow empty relative URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2704 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-07-04 20:28:55 +00:00
calvin
49aa4ec208 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2678 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-27 15:02:10 +00:00
calvin
0c3e78569c fix norming 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2676 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-23 16:21:05 +00:00
calvin
217d34bb54 use new extern check 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2675 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-22 12:30:03 +00:00
calvin
d48eb20a28 support extern filter in ignored url 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2672 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-20 14:53:23 +00:00
calvin
990d548424 don't delete from incoming queue while iterating, but after. SF #1223920 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2671 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-20 09:49:27 +00:00
calvin
ea052008a2 send the complete email address with VRFY command 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2654 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-12 21:17:10 +00:00
calvin
cc0f7cb113 always norm a URL before sending a request 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2653 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-06-12 21:16:47 +00:00
calvin
79d0850fed fix intern url of file: links 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2636 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-22 20:28:59 +00:00
calvin
fec843814c fix intern url of file: links 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2635 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-22 20:27:32 +00:00
calvin
6d8f0146d6 fix GET fallback handling 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2623 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 19:53:13 +00:00
calvin
d2107c7a6a split and fix initialization of UrlData objects 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2622 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 19:42:14 +00:00
calvin
cd32e823df fix url util call 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2621 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 19:02:52 +00:00
calvin
9666421a10 add log_url 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2620 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 19:02:40 +00:00
calvin
1fc5eb32b6 fix extern calculation and a typo 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2619 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 19:02:16 +00:00
calvin
f6f04e376e fix extern calculation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2618 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 19:01:51 +00:00
calvin
d6e56ceba2 remove lock from check_url in case of disabled threading 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2617 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 18:30:06 +00:00
calvin
55d7d5f7df use a decorator to synchronize all the checker threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2614 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-19 15:49:06 +00:00
calvin
781ad5fdc0 catch exceptions on abort 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2612 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-18 22:15:54 +00:00
calvin
fd652881fb thread name and maximum wait for finish 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2611 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-18 22:15:18 +00:00
calvin
3db056f161 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2610 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-18 22:14:45 +00:00
calvin
4c270cfe74 log message formatting 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2609 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-18 18:04:54 +00:00
calvin
c0d9e56a4d new trace option 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2601 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-17 22:06:47 +00:00
calvin
ed738d2436 cmdline param defaults to False 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2595 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-17 19:00:17 +00:00
calvin
e3a049ee27 make intern pattern of parent path 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2593 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-17 14:11:17 +00:00
calvin
d821abce85 news group info string 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2588 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-13 14:01:53 +00:00
calvin
208a7f095f added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2585 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-09 22:07:45 +00:00
calvin
1a7a771648 new intern/extern handling 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2584 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-09 22:05:21 +00:00
calvin
829e19e7ee cleanup 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2572 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-06 22:15:12 +00:00
calvin
c07f42031a ignored urls only issue an info instead of a warning 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2569 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-06 14:59:22 +00:00
calvin
bf8fd348aa better msg 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2564 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-04 13:22:10 +00:00
calvin
532a04e436 method change is an info, not a warning 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2563 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-04 13:18:19 +00:00
calvin
49a2dee338 make robots.txt deny an info instead of a warning 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2555 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-04 10:48:48 +00:00
calvin
a39f635b82 use some %r in warning strings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2554 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-05-04 10:48:10 +00:00
calvin
d2a8764d18 do not store aliases 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2550 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-27 12:48:48 +00:00
calvin
23e5212c7d do not remove userinfo from host part of url 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2544 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-26 17:53:31 +00:00
calvin
82dee6ae52 reintroduce the self.url updated, this time only for redirections but without a warning 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2542 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-26 17:51:15 +00:00
calvin
60069a8862 improved internal error msg 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2537 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-25 23:46:37 +00:00
calvin
5e6317a4bb remove old warning, and move effective url test to urlbase 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2532 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-25 18:25:32 +00:00
calvin
71407d0399 use new logger interface 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2528 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-25 14:52:38 +00:00
calvin
7925d5dd56 use collections.deque for incoming queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2518 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-04-13 21:05:52 +00:00
calvin
ba5939c7c5 allow optional empty url path 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2500 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-31 13:50:11 +00:00
calvin
bfe3f4cd4d do not add aliases that are in-progress 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2491 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 20:37:06 +00:00
calvin
5b64583904 add ignore_missing flag and do not cache already-cached aliases 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2490 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 20:26:35 +00:00
calvin
eeb6da8330 do not cache HTTP connections with pending data 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2487 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 18:47:09 +00:00
calvin
ecd7e44935 add assertion for check queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2481 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 15:52:40 +00:00
calvin
911e2eb011 fix persistent connection detection 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2477 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 13:41:22 +00:00
calvin
7429dfa209 cached connection debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2476 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 13:40:54 +00:00
calvin
a50c3ce40a use threader init param to set number of threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2469 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 11:29:49 +00:00
calvin
9cc3f978a6 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2465 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-29 08:52:40 +00:00
calvin
8e720030d1 check A RR records when no MX RR record could be found 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2461 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-28 14:59:17 +00:00
calvin
bbf96868b8 document aliases 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2458 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-18 14:21:12 +00:00
calvin
26e52022ea add aliases to cache 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2457 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-18 14:17:23 +00:00
calvin
dcd407782d no double alias reset 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2454 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-18 00:02:50 +00:00
calvin
0b0e672b4e cleanup of the fallbacks 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2453 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-16 17:34:02 +00:00
calvin
9b982062fc do not translate countries 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2434 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-12 00:43:52 +00:00
calvin
6857196b25 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2426 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-11 23:15:13 +00:00
calvin
964f313a89 fix return codes 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2424 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-11 22:39:34 +00:00
calvin
2b8ca62e2e split up the big check function 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2423 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-11 22:37:09 +00:00
calvin
ac450295ac put country info in local check method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2412 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-11 20:30:41 +00:00
calvin
131bea1a59 avoid deadlock with cache lock 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2395 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-10 22:09:41 +00:00
calvin
e89ef2c57a use own locking class 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2393 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-10 22:03:09 +00:00
calvin
4c248fce03 avoid deadlock with cache lock 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2392 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-10 22:02:52 +00:00
calvin
dc960185df only set connection debug level if CHECK logging is on debug 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2388 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 21:55:20 +00:00
calvin
7563f0edf9 fix lock class 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2387 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 21:54:56 +00:00
calvin
3373644560 make the consumer a lock object itself 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2386 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 20:28:16 +00:00
calvin
7f6d1b1022 more headers stuff 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2382 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 14:59:03 +00:00
calvin
b3c3c3eaaa split off header methods in separate file 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2381 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 13:46:58 +00:00
calvin
ca49e04730 content encoding getter method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2380 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 13:39:20 +00:00
calvin
12b726b4df strip content type mime 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2379 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 13:33:20 +00:00
calvin
f344b75c8e documentation and syntax fixes 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2378 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 10:26:30 +00:00
calvin
0d063a3b53 skip accept encoding, and list complete recursion list 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2377 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-09 08:44:45 +00:00
calvin
4972a5b2f4 add warning on decompress errors 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2375 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-08 21:04:02 +00:00
calvin
f808a03d3f no info for empty proxy 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2365 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-07 17:28:22 +00:00
calvin
e1e713036b use noproxyfor config values to ignore proxy settings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2360 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-07 13:57:01 +00:00
calvin
52d364ae54 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2359 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-07 10:38:09 +00:00
calvin
9e9e42e21e added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2358 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-07 10:23:48 +00:00
calvin
c8920096e0 make threadsafe geoip call 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2354 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-07 00:09:22 +00:00
calvin
0284f58359 log country name as info 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2352 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-03-07 00:08:48 +00:00
calvin
703da9eb65 safe decoding of non-unicode strings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2322 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-18 11:22:52 +00:00
calvin
05f5123d7c make sure infos and warnings are unicode 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2320 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-18 09:20:45 +00:00
calvin
0e12b923f6 make sure results are unicode 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2319 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-18 09:17:46 +00:00
calvin
44075c47bf clean up raise calls 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2294 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-08 14:52:50 +00:00
calvin
77cea28d42 pylint cleanups 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2292 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-08 13:43:33 +00:00
calvin
2fced4f37d documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2288 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-08 12:10:39 +00:00
calvin
d1fd01f669 remove expired connections 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2287 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-08 10:41:27 +00:00
calvin
55652cabc0 fix nt path name for network paths 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2284 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-07 15:41:53 +00:00
calvin
344bfd1103 no return from add_connection 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2278 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-06 21:20:29 +00:00
calvin
bb3b972d8f always add connections, and close timed out ones 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2277 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-06 21:20:10 +00:00
calvin
3be005186a do not print XXX message, check for connected .sock attribute and always add to cache 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2276 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-02-06 21:15:28 +00:00
calvin
1d3dd0c35b documentation added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2258 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-31 20:18:17 +00:00
calvin
4f21309b9a added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2246 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-30 22:41:59 +00:00
calvin
01ceb0aafd HTTP connection pooling 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2245 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-30 22:39:09 +00:00
calvin
9dea804d92 per-connection timeout 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2244 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-30 11:00:37 +00:00
calvin
b698701726 only release cached FTP connections 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2243 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-29 23:29:45 +00:00
calvin
e76836a607 add time import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2242 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-29 23:19:56 +00:00
calvin
372798487b start logger output before append_url is called since this can trigger logger.new_url 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2239 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-29 23:12:55 +00:00
calvin
2c40c0be3c FTP connection timeout 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2229 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-28 12:16:35 +00:00
calvin
639aeb294a return 'busy' on busy connections 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2223 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-28 01:03:16 +00:00
calvin
7e0c34a3d9 better FTP connection pooling, remove unnecessary LIST 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-28 01:02:15 +00:00
calvin
914f2440b7 syntax cleanup 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2220 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-27 21:14:17 +00:00
calvin
49cfa19c5d ignore all close errors 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2219 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-27 21:11:48 +00:00
calvin
3e79dd7e09 added and use ftpparse module 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2218 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-27 19:03:33 +00:00
calvin
5ae97df822 use python 2.4 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2209 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-24 20:29:55 +00:00
calvin
e226131e59 use new nntp argument usenetrc 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2207 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-24 19:31:13 +00:00
calvin
93e3da02b2 socket.sslerror is now a subclass of socket.error 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2206 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-24 19:27:11 +00:00
calvin
8e2b4edd49 split status message prints 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2187 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-20 22:52:58 +00:00
calvin
10e8fd9fe3 make interrupt warning a plural form 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2181 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-20 21:44:10 +00:00
calvin
86e90da338 fix reversed sslerror version check logic 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2175 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-20 12:50:38 +00:00
calvin
f40e41a5bb cleanup 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2171 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-20 09:35:46 +00:00
calvin
d030a5b054 documentation updated 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2164 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-19 15:56:48 +00:00
calvin
647d7167ee documentation syntax 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2163 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-19 15:08:02 +00:00
calvin
b008747f39 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2160 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-19 01:04:38 +00:00
calvin
46de8b6ca0 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2159 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-19 00:40:06 +00:00
calvin
ef8613f2c7 documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2157 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-19 00:08:41 +00:00
calvin
8ac75ce155 only add socket.sslerror to exception list if needed 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2156 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-18 17:04:31 +00:00
calvin
700d564be7 documentation updates 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2148 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-18 01:00:45 +00:00
calvin
be4fb033b8 print VRFY info or warning 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2136 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-11 14:34:11 +00:00
calvin
15364fc362 fix username param 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2133 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-11 13:32:47 +00:00
calvin
b3e3448f96 sort MX hosts according to preference 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2132 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-11 13:29:34 +00:00
calvin
b06f144ced updated copyright year 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2122 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2005-01-11 02:22:43 +00:00
calvin
9e843d71a7 set smtp debug 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2098 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-12-07 11:06:34 +00:00
calvin
9829268533 ignore empty filenames in filelist 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2097 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-12-07 08:16:20 +00:00
calvin
0778ef9d6d use internal file get method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2094 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-12-07 00:01:28 +00:00
calvin
1e001992ac disable recursion debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2091 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-12-06 22:38:43 +00:00
calvin
ef0d624c0d unicode fixes 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2089 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-12-06 22:34:20 +00:00
calvin
3b13dd87e2 display correct url in redirect message 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2014 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-19 13:08:26 +00:00
calvin
be5a59e9fd add dots to warnings 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2010 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-19 11:18:06 +00:00
calvin
bc89449080 add dots to info messages 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2008 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-19 11:06:31 +00:00
calvin
d2a6f4ed95 recheck robots.txt allowance after redirect 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2002 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-18 00:57:35 +00:00
calvin
89616a4bba basic robots_txt check method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2001 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-18 00:55:47 +00:00
calvin
4014012c1f basic is_extern method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1999 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-18 00:49:45 +00:00
calvin
74f1935be5 recheck extern/intern filters for redirected URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1998 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-18 00:49:33 +00:00
calvin
2765c5b116 use anchor and user info in cache key 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1987 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-15 19:16:02 +00:00
calvin
cafa412343 use base cache keys 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1986 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-15 19:15:22 +00:00
calvin
4485ab19aa only norm a changed file base_url 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1985 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-15 16:36:56 +00:00
calvin
265f942ba1 add unquoted redirectino url to info 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1980 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-14 11:02:17 +00:00
calvin
1166804eb0 use idna domain for intern patterns 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@1979 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2004-11-14 10:52:20 +00:00