aphrodite-engine

Breathing Life into Language

Aphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving the Pygmalion models to a large number of users with blazing fast speeds (thanks to FasterTransformer).

Aphrodite builds upon and integrates the exceptional work from various projects, including:

• vLLM (CacheFlow)
• FasterTransformer
• Megatron-LM
• FastChat
• SkyPilot
• OpenAI Python Library

Please note that Aphrodite is currently in active development and not yet fully functional.

Features

• Continuous Batching
• Efficient K/V management with PagedAttention
• Optimized CUDA kernels for improved inference
• Distributed inference
• Multiple decoding algorithms (e.g. parallel sampling, beam search)

Requirements

You will likely need a CUDA version of at least 11.0, and a Compute Capability of at least 7, 0. CUDA 12.0 is unsupported, so please switch to 11.8!

Linux-only

Contributing

We accept PRs! There will likely be a few typos or other errors we've failed to catch, so please let us know either via an issue or make a Pull Request.