ChatXinference

Xinference is a powerful and versatile library designed to serve LLMs, speech recognition models, and multimodal models, even on your laptop. It supports a variety of models compatible with GGML, such as chatglm, baichuan, whisper, vicuna, orca, and many others.

Overview​

Integration details​

Model features​

Setup​

Install Xinference through PyPI:

%pip install --upgrade --quiet  "xinference[all]"

Deploy Xinference Locally or in a Distributed Cluster.​

For local deployment, run xinference.

To deploy Xinference in a cluster, first start an Xinference supervisor using the xinference-supervisor. You can also use the option -p to specify the port and -H to specify the host. The default port is 8080 and the default host is 0.0.0.0.

Then, start the Xinference workers using xinference-worker on each server you want to run them on.

You can consult the README file from Xinference for more information.

Wrapper​

To use Xinference with LangChain, you need to first launch a model. You can use command line interface (CLI) to do so:

%xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0

Model uid: 7167b2b0-2a04-11ee-83f0-d29396a3f064

A model UID is returned for you to use. Now you can use Xinference with LangChain:

Installation​

The LangChain Xinference integration lives in the langchain-xinference package:

%pip install -qU langchain-xinference

Make sure you're using the latest Xinference version for structured outputs.

Instantiation​

Now we can instantiate our model object and generate chat completions:

from langchain_xinference.chat_models import ChatXinference

llm = ChatXinference(
    server_url="your_server_url", model_uid="7167b2b0-2a04-11ee-83f0-d29396a3f064"
)

llm.invoke(
    "Q: where can we visit in the capital of France?",
    config={"max_tokens": 1024},
)

Invocation​

from langchain_core.messages import HumanMessage, SystemMessage
from langchain_xinference.chat_models import ChatXinference

llm = ChatXinference(
    server_url="your_server_url", model_uid="7167b2b0-2a04-11ee-83f0-d29396a3f064"
)

system_message = "You are a helpful assistant that translates English to French. Translate the user sentence."
human_message = "I love programming."

llm.invoke([HumanMessage(content=human_message), SystemMessage(content=system_message)])

Chaining​

We can chain our model with a prompt template like so:

from langchain.prompts import PromptTemplate
from langchain_xinference.chat_models import ChatXinference

prompt = PromptTemplate(
    input=["country"], template="Q: where can we visit in the capital of {country}? A:"
)

llm = ChatXinference(
    server_url="your_server_url", model_uid="7167b2b0-2a04-11ee-83f0-d29396a3f064"
)

chain = prompt | llm
chain.invoke(input={"country": "France"})
chain.stream(input={"country": "France"})

API reference​

For detailed documentation of all ChatXinference features and configurations head to the API reference: https://github.com/TheSongg/langchain-xinference

Related​

• Chat model conceptual guide
• Chat model how-to guides