llm-huggingface

LLM plugin for models hosted by HuggingFace. This plugin is built following the llm-openrouter as a template.

Installation

First, install the LLM command-line utility.

At some point you should be able to install this plugin in the same environment as LLM by running

llm install llm-hugginface

But for now, you have to install it from sources in the the same environment as LLM. See below.

Configuration

You will need an API key from HuggingFace. You can obtain one here.

You can set that as an environment variable called HUGGINGFACE_KEY, or add it to the llm set of saved keys using:

llm keys set huggingface

Enter key: <paste key here>

Usage

To list available models, run:

llm models list

You should see a list that looks something like this:

HuggingFace: huggingface/Qwen/QwQ-32B-Preview
HuggingFace: huggingface/Qwen/Qwen2.5-Coder-32B-Instruct
HuggingFace: huggingface/google/gemma-2-2b-it
HuggingFace: huggingface/mistralai/Mixtral-8x7B-Instruct-v0.1
...

To run a prompt against a model, pass its full model ID to the -m option, like this:

llm -m huggingface/Qwen/QwQ-32B-Preview "Five spooky names for a pet tarantula"

You can set a shorter alias for a model using the llm aliases command like so:

llm aliases set QwQ huggingface/Qwen/QwQ-32B-Preview

Now you can prompt w QwQ using:

cat llm_huggingface.py | llm -m QwQ -s 'write some pytest tests for this'

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-huggingface
python3 -m venv venv
source venv/bin/activate
pip install llm

Now install the dependencies:

llm install -e '.'

Now the plugin should be available for you to try it out.