2553 commits

Author	SHA1	Message	Date
Bastian Kleineidam	ecf8753a19	Improved user-agent string similar to Google and Bing search bots.	2012-09-21 15:46:14 +02:00
Bastian Kleineidam	c274b50c50	Store lowercase URL scheme in checker class.	2012-09-21 14:35:25 +02:00
Bastian Kleineidam	0941f6ff02	Improve exception handling by using unicode.	2012-09-21 14:29:20 +02:00
Bastian Kleineidam	f46889a4af	Log timestamps in debug output.	2012-09-21 13:05:36 +02:00
Bastian Kleineidam	049882e4fe	Remove accept-encoding since some sites have wrong compression.	2012-09-20 22:39:15 +02:00
Bastian Kleineidam	7c6dce6136	Only warn non-empty site duplicates.	2012-09-20 20:39:36 +02:00
Bastian Kleineidam	a03090c20f	Optimize intern/extern pattern parsing.	2012-09-20 20:19:13 +02:00
Bastian Kleineidam	c385c35b1a	Fix ansicolor again.	2012-09-20 16:39:40 +02:00
Bastian Kleineidam	b9d234c78a	Fix wrong method name in SSL certificate check.	2012-09-20 16:28:01 +02:00
Bastian Kleineidam	bff217c58b	Never log ignored warnings.	2012-09-20 12:44:40 +02:00
Bastian Kleineidam	600b7c0e69	Fix duplicate content warning when self.size is not set yet.	2012-09-20 12:44:23 +02:00
Bastian Kleineidam	9cfee5eb5b	Improved color detection with curses.	2012-09-20 12:13:15 +02:00
Bastian Kleineidam	bc0a17c1c4	Display last modified date in the GUI.	2012-09-19 21:23:39 +02:00
Bastian Kleineidam	d37347cab0	Remove unused variable.	2012-09-19 11:08:06 +02:00
Bastian Kleineidam	18a200d85f	Fix tests.	2012-09-19 11:05:26 +02:00
Bastian Kleineidam	b8f8bdf5fc	Fix last modified formatting.	2012-09-19 10:09:19 +02:00
Bastian Kleineidam	f5fbd7666f	Remove unused import.	2012-09-19 09:39:32 +02:00
Bastian Kleineidam	75719b34f6	Updated copyright.	2012-09-19 09:17:25 +02:00
Bastian Kleineidam	71fba0f8b7	Log all valid URLs in sitemap loggers.	2012-09-19 09:17:08 +02:00
Bastian Kleineidam	9d1c90f96c	Write extra script to analyse a memory dump.	2012-09-18 16:08:31 +02:00
Bastian Kleineidam	3a352631ba	Add modified field to loggers.	2012-09-18 12:12:00 +02:00
Bastian Kleineidam	1db63227f6	Memoize file operations to minimize disk I/O.	2012-09-18 09:37:21 +02:00
Bastian Kleineidam	932a07a9cf	Added XML sitemap logger.	2012-09-18 09:16:34 +02:00
Bastian Kleineidam	4e59056ee7	Warn about duplicate URL contents.	2012-09-17 19:49:50 +02:00
Bastian Kleineidam	02a09dbb28	Add documentation.	2012-09-17 16:30:32 +02:00
Bastian Kleineidam	99bf8aa940	Updated copyright.	2012-09-17 16:09:55 +02:00
Bastian Kleineidam	cb71f483a5	Warn about too long URLs.	2012-09-17 16:00:23 +02:00
Bastian Kleineidam	03667a4ec9	Print warning tags in text output.	2012-09-17 15:29:04 +02:00
Bastian Kleineidam	1f9ee987f9	Improved terminal color detection with curses.	2012-09-17 15:24:04 +02:00
Bastian Kleineidam	6e1841cf1f	Print download and cache statistics.	2012-09-17 15:23:25 +02:00
Bastian Kleineidam	0b5b6ab37b	Automatically set --complete for graph output.	2012-09-15 15:06:29 +02:00
Bastian Kleineidam	273230d98b	Send HTTP Do-Not-Track header.	2012-09-14 22:41:38 +02:00
Bastian Kleineidam	e98f15933f	Stop checking of all output loggers have been deactivated.	2012-09-14 22:36:59 +02:00
Bastian Kleineidam	81d2c4dbd9	Improved documentation.	2012-09-14 22:26:45 +02:00
Bastian Kleineidam	86f1c74006	Close loggers properly on I/O errors.	2012-09-14 22:09:18 +02:00
Bastian Kleineidam	6730fb51ee	Allow maximum check time specification.	2012-09-03 20:17:49 +02:00
Bastian Kleineidam	a1dfaf2f91	Add missing docstring.	2012-09-02 23:37:43 +02:00
Bastian Kleineidam	21db38546c	Updated copyright.	2012-09-02 23:36:31 +02:00
Bastian Kleineidam	3baaca47a0	Add maximum number of allowed puts on URL queue.	2012-09-02 22:44:29 +02:00
Bastian Kleineidam	d8fce1ceeb	Do not sort URL queue anymore.	2012-09-02 22:32:14 +02:00
Bastian Kleineidam	7a6436f08f	Increase checked cache in URL queue.	2012-09-02 22:21:49 +02:00
Bastian Kleineidam	4c16d3e702	Make 401 unauthorized GET response a warning.	2012-08-26 11:32:17 +02:00
Bastian Kleineidam	b6d45eabe5	Code cleanup.	2012-08-24 09:46:38 +02:00
Bastian Kleineidam	ac6591a009	Recognize WML files on Windows.	2012-08-24 09:46:26 +02:00
Bastian Kleineidam	7334a9863e	Make URL properties in GUI selectable with the mouse.	2012-08-24 00:10:59 +02:00
Bastian Kleineidam	ae15d51b30	Translate more result strings.	2012-08-23 23:59:33 +02:00
Bastian Kleineidam	ce4253263c	Do not special case http->ftp redirects.	2012-08-23 23:56:36 +02:00
Bastian Kleineidam	7374068941	Remove unused import.	2012-08-23 16:46:14 +02:00
Bastian Kleineidam	73d64e50ab	Fix redirection to new scheme.	2012-08-23 16:45:24 +02:00
Bastian Kleineidam	99ab68908c	Increase the default number of checker threads.	2012-08-23 16:11:47 +02:00