- [BUGFIX] The --pause option really meant to pause between two
  subsequent requests to the _same_ host.

- [FEATURE] Add robots.txt crawl-delay support in linkchecker

- [TEST] Add test for cookie file parsing

- [BUGFIX] The URL in the log output is double normed.

- [USAGE] make a nice GUI for linkchecker

- [FEATURE] warn if overall size of page (including images/flash/etc.) is
  too big. Right now, the page size is only html content.

- [FEATURE] Option to save downloaded pages. This could also be used to
  build an internal cache, however there are already a plethora of
  caching proxies we could use for that.