 AlpinDale
|
90ceab32ff
refactor: consolidate prompt args to LLM engines
|
7 kuukautta sitten |
|
AlpinDale
|
e4ea3da1ad
fix: tensor parallel with embedding model
|
7 kuukautta sitten |
|
AlpinDale
|
f40b809d3b
allow using v2 block manager with sliding window
|
7 kuukautta sitten |
|
AlpinDale
|
2649f3f14e
aqlm works on pascal
|
7 kuukautta sitten |
|
AlpinDale
|
ac79d115b3
add guards for prefix caching, fp8, chunked, etc
|
7 kuukautta sitten |
|
AlpinDale
|
344ddaac5a
properly disable speculative decoding
|
7 kuukautta sitten |
|
AlpinDale
|
696f2cd59c
add phi3_small support with blocksparse attention
|
7 kuukautta sitten |
|
AlpinDale
|
0d15aa3ab3
fix prefix caching for block manager v2
|
7 kuukautta sitten |
|
AlpinDale
|
7d0884de9a
fix mistral v0.3 weight loading
|
7 kuukautta sitten |
|
AlpinDale
|
e8b7f53321
allow prompt token IDs in the logits processor api
|
7 kuukautta sitten |
|
AlpinDale
|
f4ea11b982
feat: initial support for activation quantization
|
7 kuukautta sitten |
 Drake
|
e1a142c179
Fix OpenAI chat completions compatibility (#559)
|
7 kuukautta sitten |
|
AlpinDale
|
5b0c11d190
support pipeline parallel pynccl groups
|
7 kuukautta sitten |
|
AlpinDale
|
f6250c5516
move dockerfiles to root; fix cpu build
|
7 kuukautta sitten |
|
AlpinDale
|
d8667fcb98
improve gptq_marlin_24 prefill performance
|
7 kuukautta sitten |
|
AlpinDale
|
eb2c5c77df
feat: enforce the max possible seqlen
|
7 kuukautta sitten |
|
AlpinDale
|
19a959a03e
prioritize user selection for attention
|
7 kuukautta sitten |
|
AlpinDale
|
c1ed789835
fix: typo in llama.py
|
7 kuukautta sitten |
|
AlpinDale
|
4e1ae004da
make mp the default distributed backend
|
7 kuukautta sitten |
|
AlpinDale
|
de62ceb18c
refactor: eliminate parallel worker per-step task scheduling overhead
|
7 kuukautta sitten |
|
AlpinDale
|
656459fd84
make fp8_e4m3 work on nvidia
|
7 kuukautta sitten |
|
AlpinDale
|
6e626b902c
fix cutlass w8a8 kernels for cuda stream
|
7 kuukautta sitten |
 AlpinDale
|
3bdeb3e116
fix: clang formatting for all kernels (#558)
|
7 kuukautta sitten |
|
AlpinDale
|
04d22bf1a9
add clang-format
|
7 kuukautta sitten |
|
AlpinDale
|
60e74e92fd
add rope_scaling arg
|
7 kuukautta sitten |
|
AlpinDale
|
b8b63eb5ca
fix head_size check for flash attention backend
|
7 kuukautta sitten |
|
AlpinDale
|
8077af0b2f
add lora support for phi
|
7 kuukautta sitten |
|
AlpinDale
|
295cfb2f39
add rope scaling for qwen2
|
7 kuukautta sitten |
|
AlpinDale
|
b7667151e5
fix scheduler being off by one for lora support
|
7 kuukautta sitten |
|
AlpinDale
|
d9d3640165
make ShardedStateLoader work with HF downloads
|
7 kuukautta sitten |
