 AlpinDale
|
3a53ff1e01
fix: raise an error for no draft token case when draft_tp>1
|
vor 5 Monaten |
|
AlpinDale
|
d638dc592d
fix: some minor typing issues in spec decode
|
vor 5 Monaten |
|
AlpinDale
|
16dff9babc
chore: enable bonus token in spec decoding for KV cache based models
|
vor 5 Monaten |
 AlpinDale
|
af43576da0
feat: add MLPSpeculator speculative decoding support (#572)
|
vor 6 Monaten |
|
AlpinDale
|
4d1e613804
chore: minor simplifications
|
vor 6 Monaten |
|
AlpinDale
|
e0886ee929
feat: add `ProposerWorkerBase` abstract class
|
vor 6 Monaten |
|
AlpinDale
|
e42d0b3455
possibly improve ngram efficiency
|
vor 6 Monaten |
|
AlpinDale
|
197a6d2c16
auto disable speculative decoding by the running queue size
|
vor 6 Monaten |
|
AlpinDale
|
ef733aee43
implement ExecuteModelData to reduce executor complexity
|
vor 6 Monaten |
|
AlpinDale
|
79901b76de
logprobs for target model (spec decoding)
|
vor 6 Monaten |
|
AlpinDale
|
9d81716bfd
[v0.5.3] Release Candidate (#388)
|
vor 9 Monaten |
