MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
🖥️ 本地推理与部署
Ollama、llama.cpp、vLLM：本地与生产环境的模型推理
全部 1047🧠 大语言模型 25🤖 AI 智能体 164🧩 应用开发平台 23📚 RAG 与知识库 46💬 聊天界面与客户端 42🖥️ 本地推理与部署 87👨💻 AI 编程 97🛠️ 开发框架与 SDK 48🔌 MCP 与工具生态 36🎨 图像生成 26🎬 视频生成 18🎙️ 语音与音频 44👁️ 计算机视觉 26📄 文档解析与抓取 24🗄️ 向量数据库 25🏋️ 训练微调与 MLOps 76🧪 评测·观测·安全 80🚪 API 网关与代理 12🔎 搜索与深度研究 13⚙️ 深度学习基础库 54🏷️ 数据与标注 31🔬 行业与科研 AI 36⚡ 效率工具 14
Large-scale LLM inference engine
Self-hosted LLM router with a managed safety net. OpenAI-compatible. BYOK. Single-workspace. Streaming. For more advanced routing choose hosted OrcaRouter
TT-NN operator library, and TT-Metalium low level kernel programming model.
Fast Multimodal LLM on Mobile Devices
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
WebAssembly binding for llama.cpp - Enabling on-browser LLM inference
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while mai
A FastAPI service for semantic text search using precomputed embeddings and advanced similarity measures, with built-in support for various file types through textract.
RAFT contains fundamental widely-used algorithms and primitives for machine learning and information retrieval. The algorithms are CUDA-accelerated and form building blocks for more easily writing high performance applic
Kubernetes AI Toolchain Operator
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
QA-Pilot is an interactive chat project that leverages online/local LLM for rapid understanding and navigation of GitHub code repository.