Domů Procházet Nápověda
Registrovat se Přihlásit se
david
/
aphrodite-engine
zrcadlo https://github.com/PygmalionAI/aphrodite-engine
1
0
Rozštěpit 0
Soubory Úkoly 0 Wiki

PygmalionAI's large-scale inference engine
pygmalion.chat

It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving the Pygmalion models to a large number of users with blazing fast speeds (thanks to vLLM's Paged Attention).

Strom: 0715cc1958
Větve Značky
aphrodite-engin...
ZIP TAR.GZ
AlpinDale 0715cc1958 fix: typo in llama modeling file před 1 rokem
aphrodite 0715cc1958 fix: typo in llama modeling file před 1 rokem
assets fefbf029c9 revert previous commit před 1 rokem
kernels 5e82533d02 upstream: add option to specify tokenizer před 1 rokem
.gitignore 07aa2a492f upstream: add option to specify tokenizer před 1 rokem
LICENSE b9bcbe5d4e Change license from Apache 2.0 to AGPLv3 před 1 rokem
README.md e75063713a fix: typo in README před 1 rokem
requirements.txt c25ee56847 chore: change epsilon variance value před 1 rokem

README.md

Breathing Life into Language

aphrodite

Aphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving the Pygmalion models to a large number of users with blazing fast speeds (thanks to FasterTransformer).

Aphrodite builds upon and integrates the exceptional work from various projects, including:

Please note that Aphrodite is currently in active development and not yet fully functional.

Features

  • Continuous Batching
  • Efficient K/V management with PagedAttention
  • Optimized CUDA kernels for improved inference
  • Distributed inference
  • Multiple decoding algorithms (e.g. parallel sampling, beam search)

Requirements

You will likely need a CUDA version of at least 11.0, and a Compute Capability of at least 7, 0. CUDA 12.0 is unsupported, so please switch to 11.8!

Linux-only

Contributing

We accept PRs! There will likely be a few typos or other errors we've failed to catch, so please let us know either via an issue or make a Pull Request.