 AlpinDale
|
aa1db50131
simplify tokenizer.py
|
9 月之前 |
|
AlpinDale
|
04c38f2b91
cache tokenizer len
|
9 月之前 |
|
AlpinDale
|
d1786645a3
fix formatting
|
9 月之前 |
|
AlpinDale
|
eed70dff76
improve detokenization performance; improve logprobs
|
9 月之前 |
 AlpinDale
|
feb5840f2a
feat: async tokenization (#374)
|
9 月之前 |
|
AlpinDale
|
b361096463
fix: tokenizer when using ray (#366)
|
9 月之前 |
|
AlpinDale
|
89c32b40ec
chore: add new imatrix quants (#320)
|
10 月之前 |
 drummerv
|
e59dd4a90d
fix: openai gguf chat template (#312)
|
10 月之前 |
|
AlpinDale
|
c2d77b1822
chore: logging refactor (#302)
|
10 月之前 |
|
AlpinDale
|
e31c6f0b45
feat: refactor modeling logic and support more models (#274)
|
10 月之前 |
|
AlpinDale
|
842912d022
feat: on-the-fly gguf conversion (#250)
|
11 月之前 |
|
AlpinDale
|
c0aac15421
feat: S-LoRA support (#222)
|
11 月之前 |
|
AlpinDale
|
e113f3e9bf
feat: spaces between special tokens (#94)
|
1 年之前 |
|
AlpinDale
|
11c01cfd0a
fix: skip special tokens
|
1 年之前 |
|
AlpinDale
|
74604eb252
fix: pylint complaints (#91)
|
1 年之前 |
|
AlpinDale
|
2c1d6a8cf2
fix: fast tokenizer latency reduction
|
1 年之前 |
|
AlpinDale
|
75c27d3e65
massive overhaul
|
1 年之前 |
|
AlpinDale
|
9328091450
Revert a revert.
|
1 年之前 |
|
AlpinDale
|
663dd09399
Revert "fix: detokenization with special tokens"
|
1 年之前 |
|
AlpinDale
|
cc1d5339dd
fix: detokenization with special tokens
|
1 年之前 |
|
AlpinDale
|
6b9561ef07
adapt TGI incremental detokenization
|
1 年之前 |
|
AlpinDale
|
45f6d9f923
initial refactor commit
|
1 年之前 |
|
AlpinDale
|
267c73c5d3
chore: minor tokenizer tweaks
|
1 年之前 |
|
AlpinDale
|
862e48176e
fix: handle special tokens
|
1 年之前 |
|
AlpinDale
|
2cdfc45a40
fix: trust_remote_code fixes
|
1 年之前 |
|
AlpinDale
|
56077f0f29
upstream: trust remote code
|
1 年之前 |
|
AlpinDale
|
5169163403
chore: add tokenizer mode for slow/fast tokenizers
|
1 年之前 |
|
AlpinDale
|
5e82533d02
upstream: add option to specify tokenizer
|
1 年之前 |
|
AlpinDale
|
07aa2a492f
upstream: add option to specify tokenizer
|
1 年之前 |
