- The binary component and library component are separated as two
packages in the same workspace.
- `lychee` is the binary component, in `lychee-bin/*`.
- `lychee-lib` is the library component, in `lychee-lib/*`.
- Users can now install only the `lychee-lib`, instead of both
components, that would require fewer dependencies and faster
compilation.
- Dependencies for each component are adjusted and updated. E.g.,
no CLI dependencies for `lychee-lib`.
- CLI tests are only moved to `lychee`, as it has nothing to do
with the library component.
- `Status::Error` is refactored to contain dedicated error enum,
`ErrorKind`.
- The motivation is to delay the formatting of errors to strings.
Note that `e.to_string()` is not necessarily cheap (though
trivial in many cases). The formatting is no delayed until the
error is needed to be displayed to users. So in some cases, if
the error is never used, it means that it won't be formatted at
all.
- Replaced `regex` based matching with one of the following:
- Simple string equality test in the case of 'false positivie'.
- URL parsing based test, in the case of extracting repository and
user name for GitHub links.
- Either cases would be much more efficient than `regex` based
matching. First, there's no need to construct a state machine for
regex. Second, URL is already verified and parsed on its creation,
and extracting its components is fairly cheap. Also, this removes
the dependency on `lazy-static` in `lychee-lib`.
- `types` module now has a sub-directory, and its components are now
separated into their own modules (in that sub-directory).
- `lychee-lib::test_utils` module is only compiled for tests.
- `wiremock` is moved to `dev-dependency` as it's only needed for
`test` modules.
- Dependencies are listed in alphabetical order.
- Imports are organized in the following fashion:
- Imports from `std`
- Imports from 3rd-party crates, and `lychee-lib`.
- Imports from `crate::*` or `super::*`.
- No glob import.
- I followed suggestion from `cargo clippy`, with `clippy::all` and
`clippy:pedantic`.
Co-authored-by: Lucius Hu <lebensterben@users.noreply.github.com>
|
||
|---|---|---|
| .github/workflows | ||
| assets | ||
| fixtures | ||
| lychee-bin | ||
| lychee-lib | ||
| .dockerignore | ||
| .editorconfig | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| Dockerfile | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| lychee.example.toml | ||
| Makefile | ||
| README.md | ||
| rust-toolchain | ||
| TROUBLESHOOTING.md | ||
⚡ A fast, async, resource-friendly link checker written in Rust.\ Finds broken hyperlinks and mail addresses inside Markdown, HTML, reStructuredText, or any other text file or website!
Available as a CLI utility and as a GitHub Action: lycheeverse/lychee-action.
Features
This comparison is made on a best-effort basis. Please create a PR to fix outdated information. use
| lychee | awesome_bot | muffet | broken-link-checker | linkinator | linkchecker | markdown-link-check | fink | |
|---|---|---|---|---|---|---|---|---|
| Language | Rust | Ruby | Go | JS | TypeScript | Python | JS | PHP |
| Async/Parallel | ||||||||
| JSON output | ||||||||
| Static binary | ️ |
|||||||
| Markdown files | ️ |
|||||||
| HTML files | ||||||||
| Text files | ||||||||
| Website support | ||||||||
| Chunked encodings | ||||||||
| GZIP compression | ||||||||
| Basic Auth | ||||||||
| Custom user agent | ||||||||
| Relative URLs | ||||||||
| Skip relative URLs | ||||||||
| Include patterns | ||||||||
| Exclude patterns | ||||||||
| Handle redirects | ||||||||
| Ignore insecure SSL | ||||||||
| File globbing | ||||||||
| Limit scheme | ||||||||
| Custom headers | ||||||||
| Summary | ||||||||
HEAD requests |
||||||||
| Colored output | ||||||||
| Filter status code | ||||||||
| Custom timeout | ||||||||
| E-mail links | ||||||||
| Progress bar | ||||||||
| Retry and backoff | ||||||||
| Skip private domains | ||||||||
| Use as library | ||||||||
| Quiet mode | ||||||||
| Config file | ||||||||
| Recursion | ||||||||
| Amazing lychee logo |
1 Other machine-readable formats like CSV are supported.
Contributing to lychee
We'd be thankful for any contribution.
We try to keep the issue-tracker up-to-date so you can quickly find a task to work on.
Try one of these links to get started:
Using the Commandline Client
You can run lychee directly from the commandline.
Installation
Using cargo
cargo install lychee
Using the official Docker image
docker pull lycheeverse/lychee
Using pre-built binaries
We provide binaries for Linux, macOS, and Windows for every release.
You can download them from the releases page.
Commandline usage
Run it inside a repository with a README.md:
lychee
You can also specify various types of inputs:
# check links on a website:
lychee https://endler.dev/
# check links in a remote file:
lychee https://raw.githubusercontent.com/lycheeverse/lychee/master/README.md
# check links in local file(s):
lychee README.md
lychee test.html info.txt
# check links in local files (by shell glob):
lychee ~/projects/*/README.md
# check links in local files (lychee supports advanced globbing and ~ expansion):
lychee "~/projects/big_project/**/README.*"
# ignore case when globbing and check result for each link:
lychee --glob-ignore-case --verbose "~/projects/**/[r]eadme.*"
GitHub token
Optionally, to avoid getting rate-limited while checking GitHub links, you can
set an environment variable with your Github token like so GITHUB_TOKEN=xxxx,
or use the --github-token CLI option. It can also be set in the config file.
The token can be generated in your GitHub account settings page. A personal token with no extra permissions is enough to be able to check public repos links.
Commandline Parameters
There is an extensive list of commandline parameters to customize the behavior, see below for a full list.
USAGE:
lychee [FLAGS] [OPTIONS] [--] [inputs]...
FLAGS:
-E, --exclude-all-private Exclude all private IPs from checking. Equivalent to `--exclude-private --exclude-link-
local --exclude-loopback`
--exclude-link-local Exclude link-local IP address range from checking
--exclude-loopback Exclude loopback IP address range from checking
--exclude-mail Exclude all mail addresses from checking
--exclude-private Exclude private IP address ranges from checking
--glob-ignore-case Ignore case when expanding filesystem path glob inputs
--help Prints help information
-i, --insecure Proceed for server connections considered insecure (invalid TLS)
-n, --no-progress Do not show progress bar. This is recommended for non-interactive shells (e.g. for
continuous integration)
--skip-missing Skip missing input files (default is to error if they don't exist)
-V, --version Prints version information
-v, --verbose Verbose program output
OPTIONS:
-a, --accept <accept> Comma-separated list of accepted status codes for valid links
-b, --base-url <base-url> Base URL to check relative URLs
--basic-auth <basic-auth> Basic authentication support. E.g. `username:password`
-c, --config <config-file> Configuration file to use [default: ./lychee.toml]
--exclude <exclude>... Exclude URLs from checking (supports regex)
-f, --format <format> Output file format of status report (json, string) [default: string]
--github-token <github-token> GitHub API token to use when checking github.com links, to avoid rate
limiting [env: GITHUB_TOKEN=]
-h, --headers <headers>... Custom request headers
--include <include>... URLs to check (supports regex). Has preference over all excludes
--max-concurrency <max-concurrency> Maximum number of concurrent network requests [default: 128]
-m, --max-redirects <max-redirects> Maximum number of allowed redirects [default: 10]
-X, --method <method> Request method [default: get]
-o, --output <output> Output file of status report
-s, --scheme <scheme> Only test links with the given scheme (e.g. https)
-T, --threads <threads> Number of threads to utilize. Defaults to number of cores available to
the system
-t, --timeout <timeout> Website timeout from connect to response finished [default: 20]
-u, --user-agent <user-agent> User agent [default: lychee/0.6.0]
ARGS:
<inputs>... The inputs (where to get links to check from). These can be: files (e.g. `README.md`), file globs
(e.g. `"~/git/*/README.md"`), remote URLs (e.g. `https://example.org/README.md`) or standard
input (`-`). Prefix with `--` to separate inputs from options that allow multiple arguments
[default: README.md]
Exit codes
0for success (all links checked successfully or excluded/skipped as configured)1for missing inputs and any unexpected runtime failures or config errors2for link check failures (if any non-excluded link failed the check)
Library usage
You can use lychee as a library for your own projects. Here is a "hello world" example:
use lychee_lib::Result;
#[tokio::main]
async fn main() -> Result<()> {
let response = lychee_lib::check("https://github.com/lycheeverse/lychee").await?;
println!("{}", response);
Ok(())
}
This is equivalent to the following snippet, in which we build our own client:
use lychee_lib::{ClientBuilder, Result, Status};
#[tokio::main]
async fn main() -> Result<()> {
let client = ClientBuilder::default().build()?;
let response = client.check("https://github.com/lycheeverse/lychee").await?;
assert!(response.status().is_success());
Ok(())
}
The client builder is very customizable:
let client = lychee_lib::ClientBuilder::default()
.includes(includes)
.excludes(excludes)
.max_redirects(cfg.max_redirects)
.user_agent(cfg.user_agent)
.allow_insecure(cfg.insecure)
.custom_headers(headers)
.method(method)
.timeout(timeout)
.verbose(cfg.verbose)
.github_token(cfg.github_token)
.scheme(cfg.scheme)
.accepted(accepted)
.build()?;
All options that you set will be used for all link checks. See the builder documentation for all options.
GitHub Action usage
A GitHub Action that uses lychee is available as a separate repository: lycheeverse/lychee-action which includes usage instructions.
Troubleshooting and workarounds
We collect a list of common workarounds for various websites in our troubleshooting guide.
Users
- https://github.com/pawroman/links
- https://github.com/analysis-tools-dev/static-analysis
- https://github.com/analysis-tools-dev/dynamic-analysis
- https://github.com/mre/idiomatic-rust
- https://github.com/lycheeverse/lychee (yes, the lychee docs are checked with lychee 🤯)
If you are using lychee for your project, we'd be delighted to hear about it.
License
lychee is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Credits
The first prototype of lychee was built in episode 10 of Hello Rust. Thanks to all Github- and Patreon sponsors for supporting the development since the beginning. Also, thanks to all the great contributors who have since made this project more mature.
