PygmalionAI's large-scale inference engine
pygmalion.chat

It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving the Pygmalion models to a large number of users with blazing fast speeds (thanks to vLLM's Paged Attention).

AlpinDale 926ccfd387 exponent is 1.0 by default hace 4 meses
.github a03e0e2ea4 ci: exclude cu118 and cu121 from build and add py_limited_api (#639) hace 4 meses
aphrodite 926ccfd387 exponent is 1.0 by default hace 4 meses
assets b3df2351c8 readme: update with bsz1 graph hace 10 meses
cmake f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
docker f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
docs a0e446a17d feat: initial encoder-decoder support with BART model (#633) hace 4 meses
examples a0e446a17d feat: initial encoder-decoder support with BART model (#633) hace 4 meses
kernels a401f8e05d feat: per-tensor token epilogue kernels (#630) hace 4 meses
tests f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
.clang-format f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
.dockerignore f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
.env f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
.gitignore f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
CMakeLists.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
CODE_OF_CONDUCT.md e7ea38f243 chore: add contribution guidelines + Code of Conduct (#507) hace 6 meses
CONTRIBUTING.md e7ea38f243 chore: add contribution guidelines + Code of Conduct (#507) hace 6 meses
Dockerfile f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.cpu f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.neuron f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.openvino f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.ppc64le f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.rocm f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.tpu f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
Dockerfile.xpu f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
LICENSE 5adcb33e14 Revert license back to AGPLv3 (#38) hace 1 año
MANIFEST.in f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
README.md ba848b00f3 readme: fix model name typo (#627) hace 4 meses
build_wheel.sh f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
config.yaml f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
docker-compose.yml f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
entrypoint.sh f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
env.py e42a78381a feat: switch from pylint to ruff (#322) hace 9 meses
environment.yaml f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
formatting.sh f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
mypy.ini 9d81716bfd [v0.5.3] Release Candidate (#388) hace 8 meses
patch_xformers.rocm.sh 13d850334e fix: navi support (#283) hace 10 meses
pyproject.toml f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-adag.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-build.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-common.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-cpu.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-cuda.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-dev.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-lint.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-neuron.txt 9d81716bfd [v0.5.3] Release Candidate (#388) hace 8 meses
requirements-openvino.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-rocm.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-test.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-tpu.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
requirements-xpu.txt f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
runtime.sh cbe37e8b18 fix: speed up cuda home detection (#288) hace 10 meses
setup.py f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses
update-runtime.sh f1d0b77c92 [0.6.0] Release Candidate (#481) hace 4 meses

README.md

Breathing Life into Language

aphrodite

Aphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving Hugging Face-compatible models to a large number of users with blazing fast speeds (thanks to vLLM's Paged Attention).

Aphrodite builds upon and integrates the exceptional work from various projects.

The compute necessary for Aphrodite's development is provided by Arc Compute.

🔥 News

(09/2024) v0.6.0 is released, with huge throughput improvements, many new quant formats (including fp8 and llm-compressor), asymmetric tensor parallel, pipeline parallel and more! Please check out the exhaustive documentation for the User and Developer guides.

Features

  • Continuous Batching
  • Efficient K/V management with PagedAttention from vLLM
  • Optimized CUDA kernels for improved inference
  • Quantization support via AQLM, AWQ, Bitsandbytes, GGUF, GPTQ, QuIP#, Smoothquant+, SqueezeLLM, Marlin, FP4, FP6, FP8, FP12
  • Distributed inference
  • 8-bit KV Cache for higher context lengths and throughput, at both FP8 E5M3 and E4M3 formats.

Quickstart

Install the engine:

pip install -U aphrodite-engine==0.6.0

Then launch a model:

aphrodite run meta-llama/Meta-Llama-3.1-8B-Instruct

This will create a OpenAI-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports OpenAI, such as SillyTavern.

Please refer to the documentation for the full list of arguments and flags you can pass to the engine.

You can play around with the engine in the demo here:

Open In Colab

Docker

Additionally, we provide a Docker image for easy deployment. Here's a basic command to get you started:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    #--env "CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7" \
    -p 2242:2242 \
    --ipc=host \
    alpindale/aphrodite-openai:latest \
    --model NousResearch/Meta-Llama-3.1-8B-Instruct \
    --tensor-parallel-size 8 \
    --api-keys "sk-empty"

This will pull the Aphrodite Engine image (~8GiB download), and launch the engine with the Llama-3.1-8B-Instruct model at port 2242.

Requirements

  • Operating System: Linux (or WSL for Windows)
  • Python: 3.8 to 3.12

For windows users, it's recommended to use tabbyAPI instead, if you do not need batching support.

Build Requirements:

  • CUDA >= 11

For supported devices, see here. Generally speaking, all semi-modern GPUs are supported - down to Pascal (GTX 10xx, P40, etc.) We also support AMD GPUs, Intel CPUs and GPUs, Google TPU, and AWS Inferentia.

Notes

  1. By design, Aphrodite takes up 90% of your GPU's VRAM. If you're not serving an LLM at scale, you may want to limit the amount of memory it takes up. You can do this in the API example by launching the server with the --gpu-memory-utilization 0.6 (0.6 means 60%).

  2. You can view the full list of commands by running aphrodite run --help.

Acknowledgements

Aphrodite Engine would have not been possible without the phenomenal work of other open-source projects. Credits go to:

Contributing

Everyone is welcome to contribute. You can support the project by opening Pull Requests for new features, fixes, or general UX improvements.