2003-05-05 06:36:42 +00:00
|
|
|
possible Python 2.3 improvements (ie needs Python >= 2.3)
|
2003-01-05 02:01:50 +00:00
|
|
|
- get rid of timeoutsocket.py, the default socket has timeouts
|
|
|
|
|
- use optparse instead of getopt with more flexible commandline help
|
|
|
|
|
- replace the debug() function with the logging module
|
|
|
|
|
we'll see how we can insert multiple debug levels into this thing
|
|
|
|
|
- use Bool object type
|
2003-04-29 14:55:42 +00:00
|
|
|
- get rid of the patched robotparser.py
|
|
|
|
|
|
|
|
|
|
include some web check and/or spider features:
|
|
|
|
|
- warn if overall size of page (including images/flash/etc.) is too big
|
|
|
|
|
- save downloaded pages
|
