llm/docs/plugins/directory.md

(plugin-directory)=

Plugin directory

The following plugins are available for LLM. Here's {ref}how to install them <installing-plugins>.

Local models

These plugins all help you run LLMs directly on your own computer:

• llm-gguf uses llama.cpp to run models published in the GGUF format.
• llm-mlc can run local models released by the MLC project, including models that can take advantage of the GPU on Apple Silicon M1/M2 devices.
• llm-gpt4all adds support for various models released by the GPT4All project that are optimized to run locally on your own machine. These models include versions of Vicuna, Orca, Falcon and MPT - here's a full list of models.
• llm-mpt30b adds support for the MPT-30B local model.
• llm-ollama adds support for local models run using Ollama.
• llm-llamafile adds support for local models that are running locally using llamafile.

Remote APIs

These plugins can be used to interact with remotely hosted models via their API:

• llm-mistral adds support for Mistral AI's language and embedding models.
• llm-gemini adds support for Google's Gemini models.
• llm-anthropic supports Anthropic's Claude 3 family, 3.5 Sonnet and beyond.
• llm-command-r supports Cohere's Command R and Command R Plus API models.
• llm-reka supports the Reka family of models via their API.
• llm-perplexity by Alexandru Geana supports the Perplexity Labs API models, including llama-3-sonar-large-32k-online which can search for things online and llama-3-70b-instruct.
• llm-groq by Moritz Angermann provides access to fast models hosted by Groq.
• llm-grok by Benedikt Hiepler providing access to Grok model using the xAI API Grok.
• llm-anyscale-endpoints supports models hosted on the Anyscale Endpoints platform, including Llama 2 70B.
• llm-replicate adds support for remote models hosted on Replicate, including Llama 2 from Meta AI.
• llm-fireworks supports models hosted by Fireworks AI.
• llm-palm adds support for Google's PaLM 2 model.
• llm-openrouter provides access to models hosted on OpenRouter.
• llm-cohere by Alistair Shepherd provides cohere-generate and cohere-summarize API models, powered by Cohere.
• llm-bedrock adds support for Nova by Amazon via Amazon Bedrock.
• llm-bedrock-anthropic by Sean Blakey adds support for Claude and Claude Instant by Anthropic via Amazon Bedrock.
• llm-bedrock-meta by Fabian Labat adds support for Llama 2 and Llama 3 by Meta via Amazon Bedrock.
• llm-together adds support for the Together AI extensive family of hosted openly licensed models.
• llm-deepseek adds support for the DeepSeek's DeepSeek-Chat and DeepSeek-Coder models.
• llm-lambda-labs provides access to models hosted by Lambda Labs, including the Nous Hermes 3 series.
• llm-venice provides access to uncensored models hosted by privacy-focused Venice AI, including Llama 3.1 405B.

If an API model host provides an OpenAI-compatible API you can also configure LLM to talk to it without needing an extra plugin.

Embedding models

{ref}Embedding models <embeddings> are models that can be used to generate and store embedding vectors for text.

• llm-sentence-transformers adds support for embeddings using the sentence-transformers library, which provides access to a wide range of embedding models.
• llm-clip provides the CLIP model, which can be used to embed images and text in the same vector space, enabling text search against images. See Build an image search engine with llm-clip for more on this plugin.
• llm-embed-jina provides Jina AI's 8K text embedding models.
• llm-embed-onnx provides seven embedding models that can be executed using the ONNX model framework.

Extra commands

• llm-cmd accepts a prompt for a shell command, runs that prompt and populates the result in your shell so you can review it, edit it and then hit <enter> to execute or ctrl+c to cancel.
• llm-cmd-comp provides a key binding for your shell that will launch a chat to build the command. When ready, hit <enter> and it will go right back into your shell command line, so you can run it.
• llm-python adds a llm python command for running a Python interpreter in the same virtual environment as LLM. This is useful for debugging, and also provides a convenient way to interact with the LLM {ref}python-api if you installed LLM using Homebrew or pipx.
• llm-cluster adds a llm cluster command for calculating clusters for a collection of embeddings. Calculated clusters can then be passed to a Large Language Model to generate a summary description.
• llm-jq lets you pipe in JSON data and a prompt describing a jq program, then executes the generated program against the JSON.

Just for fun

• llm-markov adds a simple model that generates output using a Markov chain. This example is used in the tutorial Writing a plugin to support a new model.