f740a5cbbd
* WIP fragments: schema plus reading but not yet writing, refs #617 * Unique index on fragments.alias, refs #617 * Fragments are now persisted, added basic CLI commands * Fragment aliases work now, refs #617 * Improved help for -f/--fragment * Support fragment hash as well * Documentation for fragments * Better non-JSON display of llm fragments list * llm fragments -q search option * _truncate_string is now truncate_string * Use condense_json to avoid duplicate data in JSON in DB, refs #617 * Follow up to 3 redirects for fragments * Python API docs for fragments= and system_fragments= * Fragment aliases cannot contain a : - this is to ensure we can add custom fragment loaders later on, refs https://github.com/simonw/llm/pull/859#issuecomment-2761534692 * Use template fragments when running prompts * llm fragments show command plus llm fragments group tests * Tests for fragments family of commands * Test for --save with fragments * Add fragments tables to docs/logging.md * Slightly better llm fragments --help * Handle fragments in past conversations correctly * Hint at llm prompt --help in llm --help, closes #868 * llm logs -f filter plus show fragments in llm logs --json * Include prompt and system fragments in llm logs -s * llm logs markdown fragment output and tests, refs #617 |
||
|---|---|---|
| .github | ||
| docs | ||
| llm | ||
| tests | ||
| .gitignore | ||
| .readthedocs.yaml | ||
| Justfile | ||
| LICENSE | ||
| MANIFEST.in | ||
| mypy.ini | ||
| pytest.ini | ||
| README.md | ||
| ruff.toml | ||
| setup.py | ||
LLM
A CLI utility and Python library for interacting with Large Language Models, both via remote APIs and models that can be installed and run on your own machine.
Run prompts from the command-line, store the results in SQLite, generate embeddings and more.
Consult the LLM plugins directory for plugins that provide access to remote and local models.
Full documentation: llm.datasette.io
Background on this project:
- llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs
- The LLM CLI tool now supports self-hosted language models via plugins
- Accessing Llama 2 from the command-line with the llm-replicate plugin
- Run Llama 2 on your own Mac using LLM and Homebrew
- Catching up on the weird world of LLMs
- LLM now provides tools for working with embeddings
- Build an image search engine with llm-clip, chat with models with llm chat
- Many options for running Mistral models in your terminal using LLM
Installation
Install this tool using pip:
pip install llm
Or using Homebrew:
brew install llm
Detailed installation instructions.
Getting started
If you have an OpenAI API key you can get started using the OpenAI models right away.
As an alternative to OpenAI, you can install plugins to access models by other providers, including models that can be installed and run on your own device.
Save your OpenAI API key like this:
llm keys set openai
This will prompt you for your key like so:
Enter key: <paste here>
Now that you've saved a key you can run a prompt like this:
llm "Five cute names for a pet penguin"
1. Waddles
2. Pebbles
3. Bubbles
4. Flappy
5. Chilly
Read the usage instructions for more.
Installing a model that runs on your own machine
LLM plugins can add support for alternative models, including models that run on your own machine.
To download and run Mistral 7B Instruct locally, you can install the llm-gpt4all plugin:
llm install llm-gpt4all
Then run this command to see which models it makes available:
llm models
gpt4all: all-MiniLM-L6-v2-f16 - SBert, 43.76MB download, needs 1GB RAM
gpt4all: orca-mini-3b-gguf2-q4_0 - Mini Orca (Small), 1.84GB download, needs 4GB RAM
gpt4all: mistral-7b-instruct-v0 - Mistral Instruct, 3.83GB download, needs 8GB RAM
...
Each model file will be downloaded once the first time you use it. Try Mistral out like this:
llm -m mistral-7b-instruct-v0 'difference between a pelican and a walrus'
You can also start a chat session with the model using the llm chat command:
llm chat -m mistral-7b-instruct-v0
Chatting with mistral-7b-instruct-v0
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
>
Using a system prompt
You can use the -s/--system option to set a system prompt, providing instructions for processing other input to the tool.
To describe how the code in a file works, try this:
cat mycode.py | llm -s "Explain this code"
Help
For help, run:
llm --help
You can also use:
python -m llm --help