Hopiu/lychee: ⚡ Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!

mirror of https://github.com/Hopiu/lychee.git synced 2026-03-22 07:20:25 +00:00

⚡ Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!

Find a file

Paweł Romanowski 96a92deffd Use more descriptive name for cargo test action		2020-10-16 14:47:54 +02:00
.github/workflows	Use more descriptive name for cargo test action	2020-10-16 14:47:54 +02:00
assets	Add logging and proper URL parsing	2020-08-07 19:00:21 +02:00
fixtures	Add mail	2020-08-23 23:19:21 +02:00
src	Introduce cargo fmt and clippy checks, fix all clippy warnings	2020-10-16 14:35:38 +02:00
.gitignore	Extract from hello-rust/show repository	2020-08-05 00:32:37 +02:00
Cargo.lock	Add progress bar	2020-10-10 00:39:06 -04:00
Cargo.toml	Add progress bar	2020-10-10 00:39:06 -04:00
README.md	Mention progress bar support by @xiaochuanyu	2020-10-15 02:13:39 +02:00

README.md

...because who says I can't write yet another link checker?

What?

This thing was created from Hello Rust Episode 10. It's a link checker that treats Github links specially by using a GITHUB_TOKEN to avoid getting blocked by the rate limiter.

TODO: Add screenshots here

Why?

The existing link checkers were not flexible enough for my use-case. lychee runs all requests fully asynchronously and has a low memory/CPU footprint.

lychee can...

handle links inside Markdown, HTML, and other documents
handle chunked encodings
handle gzip compression
fake user agents (required for some firewalls)
skip non-links like anchors or relative URLs
exclude some websites with regular expressions
handle a configurable number of redirects
disguise as a different user agent (like curl)
optionally ignore SSL certificate errors (--insecure)
check multiple files at once (supports globbing)
support checking links from any website URL
limit scheme (e.g. only check HTTPS links with "https")
accept custom headers (e.g. for cases like https://github.com/rust-lang/crates.io/issues/788)
show final summary/statistics
optionally use HEAD requests instead of GET
show colored output
filter based on status codes (https://github.com/tcort/markdown-link-check/issues/94) (e.g. --accept 200,204)
accept a request timeout (--timeout) in seconds. Default is 20s. Set to 0 for no timeout.
check e-mail links using check-if-mail-exists
show the progress interactively with progress bar and in-flight requests (--progress) by @xiaochuanyu

SOON:

automatically retry and backoff
check relative (base-url to set project root)
usable as a library (https://github.com/raviqqe/liche/issues/13)
exclude private domains (a5102b0bf9/url_checker.go)
recursion
extended statistics: request latency
use colored output (https://crates.io/crates/colored)

Users

SOON: https://github.com/analysis-tools-dev/static-analysis

How?

cargo install lychee

Set an environment variable with your token like so GITHUB_TOKEN=xxxx.

Run it inside a repository with a README.md or specify a different Markdown file with

lychee <yourfile>

Comparison

Collecting other link checkers here to crush them in comparison. :P

Thanks

...to my Github sponsors and Patreon sponsors for supporting these projects. If you want to help out as well, go here.