1. Project Overview

Xorbits Inference: model inference, easy to use

Xorbits Inference (Xinference) is a high-performance and feature-rich distributed inference framework. It can be used for the inference of various models, including large language models (LLM), speech recognition models, and multimodal models, etc. With Xorbits Inference, you can easily deploy your own models or built-in advanced open-source models. Whether you are a researcher, developer, or data scientist, you can use Xorbits Inference to explore more possibilities with the latest AI models.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

Why choose Xinference

Features	Xinference	FastChat	OpenLLM	RayLLM
Compatible with OpenAI's RESTful API	✅	✅	✅	✅
vLLM integration	✅	✅	✅	✅
More inference engines (GGML, TensorRT)	✅	❌	✅	✅
More platform support (CPU, Metal)	✅	✅	❌	❌
Distributed cluster deployment	✅	❌	❌	✅
Image models (text-to-image)	✅	✅	❌	❌
Text embedding models	✅	❌	❌	❌
Multimodal models	✅	❌	❌	❌
Speech recognition models	✅	❌	❌	❌
More OpenAI functions (function calls)	✅	❌	❌	❌

5. Tech Stack

Integration

• FastGPT: an open-source AI knowledge base construction platform based on LLM large models. It provides out-of-the-box data processing, model calls, RAG retrieval, and visualization of AI workflow orchestration, etc. It helps you easily implement complex question-answering scenarios.
• Dify: an LLMOps platform that covers large language model development, deployment, maintenance, and optimization.
• RAGFlow: an open-source RAG engine based on deep document understanding.
• MaxKB: MaxKB = Max Knowledge Base, an open-source knowledge base question-answering system based on large language models and RAG. It is widely used in intelligent customer service, enterprise internal knowledge bases, academic research, and education, etc.
• Chatbox: a desktop client that supports advanced large language models, supporting Windows, Mac, and Linux.