Hopiu/lychee
1
0
Fork 0
mirror of https://github.com/Hopiu/lychee.git synced 2026-04-18 20:21:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
⚡ Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!
83 commits 1 branch 0 tags 5.1 MiB
Rust 98.6%
Dockerfile 0.7%
Nix 0.3%
Makefile 0.3%
Shell 0.1%
Find a file
Paweł Romanowski 69e18785f9 Implement exclude private URLs feature 
The exclusion is currently based on IP addresses, as specified by inputs.
We support IPv4 and IPv6, where possible using the current stable stdlib
(as of Rust 1.47.0).

Note that we could go one step further and resolve all URIs using DNS
and then exclude-filter the private IPs.
2020-10-16 13:59:42 +02:00
.github/workflows remove names 2020-10-14 19:19:14 -04:00
assets Add logging and proper URL parsing 2020-08-07 19:00:21 +02:00
fixtures Add mail 2020-08-23 23:19:21 +02:00
src Implement exclude private URLs feature 2020-10-16 13:59:42 +02:00
.gitignore Extract from hello-rust/show repository 2020-08-05 00:32:37 +02:00
Cargo.lock Add progress bar 2020-10-10 00:39:06 -04:00
Cargo.toml Add progress bar 2020-10-10 00:39:06 -04:00
README.md Mention progress bar support by @xiaochuanyu 2020-10-15 02:13:39 +02:00

README.md

lychee

Rust

...because who says I can't write yet another link checker?

What?

This thing was created from Hello Rust Episode 10. It's a link checker that treats Github links specially by using a GITHUB_TOKEN to avoid getting blocked by the rate limiter.

TODO: Add screenshots here

Why?

The existing link checkers were not flexible enough for my use-case. lychee runs all requests fully asynchronously and has a low memory/CPU footprint.

lychee can...

  • handle links inside Markdown, HTML, and other documents
  • handle chunked encodings
  • handle gzip compression
  • fake user agents (required for some firewalls)
  • skip non-links like anchors or relative URLs
  • exclude some websites with regular expressions
  • handle a configurable number of redirects
  • disguise as a different user agent (like curl)
  • optionally ignore SSL certificate errors (--insecure)
  • check multiple files at once (supports globbing)
  • support checking links from any website URL
  • limit scheme (e.g. only check HTTPS links with "https")
  • accept custom headers (e.g. for cases like https://github.com/rust-lang/crates.io/issues/788)
  • show final summary/statistics
  • optionally use HEAD requests instead of GET
  • show colored output
  • filter based on status codes (https://github.com/tcort/markdown-link-check/issues/94) (e.g. --accept 200,204)
  • accept a request timeout (--timeout) in seconds. Default is 20s. Set to 0 for no timeout.
  • check e-mail links using check-if-mail-exists
  • show the progress interactively with progress bar and in-flight requests (--progress) by @xiaochuanyu

SOON:

Users

How?

cargo install lychee

Set an environment variable with your token like so GITHUB_TOKEN=xxxx.

Run it inside a repository with a README.md or specify a different Markdown file with

lychee <yourfile>

Comparison

Collecting other link checkers here to crush them in comparison. :P

Thanks

...to my Github sponsors and Patreon sponsors for supporting these projects. If you want to help out as well, go here.