 AlpinDale
|
a4cbcfe59f
feat: disable logprob serialization to CPU for spec decode
|
vor 5 Monaten |
|
AlpinDale
|
3a53ff1e01
fix: raise an error for no draft token case when draft_tp>1
|
vor 5 Monaten |
|
AlpinDale
|
2c653a2268
fix: make speculative decoding work with per-request seed
|
vor 5 Monaten |
|
AlpinDale
|
16dff9babc
chore: enable bonus token in spec decoding for KV cache based models
|
vor 5 Monaten |
 AlpinDale
|
d9f4c36edd
feat: Medusa speculative decoding support (#590)
|
vor 5 Monaten |
|
AlpinDale
|
dd378ea063
feat: MLPSpeculator with tensor parallel
|
vor 6 Monaten |
|
AlpinDale
|
7253e9052d
feat: integrate typical acceptance sampling for spec decoding
|
vor 6 Monaten |
|
AlpinDale
|
cdff8e89f9
feat: introduce `DraftModelRunner`
|
vor 6 Monaten |
|
AlpinDale
|
b6ff0623a6
chore: clean up branding
|
vor 6 Monaten |
|
AlpinDale
|
abbb730607
feat: support draft model on different tensor parallel size
|
vor 6 Monaten |
|
AlpinDale
|
af43576da0
feat: add MLPSpeculator speculative decoding support (#572)
|
vor 6 Monaten |
|
AlpinDale
|
313e6e1ec7
feat: add typical acceptance sampling
|
vor 6 Monaten |
|
AlpinDale
|
4d1e613804
chore: minor simplifications
|
vor 6 Monaten |
|
AlpinDale
|
e0886ee929
feat: add `ProposerWorkerBase` abstract class
|
vor 6 Monaten |
|
AlpinDale
|
344ddaac5a
properly disable speculative decoding
|
vor 6 Monaten |
|
AlpinDale
|
5b0c11d190
support pipeline parallel pynccl groups
|
vor 6 Monaten |
|
AlpinDale
|
de62ceb18c
refactor: eliminate parallel worker per-step task scheduling overhead
|
vor 6 Monaten |
|
AlpinDale
|
236be273e5
feat: tensor parallel speculative decoding (#554)
|
vor 6 Monaten |
|
AlpinDale
|
197a6d2c16
auto disable speculative decoding by the running queue size
|
vor 6 Monaten |
|
AlpinDale
|
438f5bdce9
fix ngrams
|
vor 6 Monaten |
|
AlpinDale
|
ef733aee43
implement ExecuteModelData to reduce executor complexity
|
vor 6 Monaten |
|
AlpinDale
|
79901b76de
logprobs for target model (spec decoding)
|
vor 6 Monaten |
|
AlpinDale
|
723c6acb84
re-add ngram speculative decoding
|
vor 6 Monaten |
|
AlpinDale
|
7bcf4c3fc9
centralize gpu worker construction
|
vor 6 Monaten |
|
AlpinDale
|
9d81716bfd
[v0.5.3] Release Candidate (#388)
|
vor 9 Monaten |
|
AlpinDale
|
f8dfac6372
chore: attention refactor and upstream sync apr01 (#365)
|
vor 10 Monaten |
