 SueJane
|
3f1b4d38e7
Fix: check the type of max_seqlen_k instead of checking max_seqlen twice (#1127)
|
vor 5 Monaten |
 Markus Krimmel
|
6bbc532388
fix: cast the alibi slopes to torch.float32 (#846)
|
vor 9 Monaten |
 Tri Dao
|
a190df011c
Add window_size option to ParallelMHA
|
vor 11 Monaten |
|
Tri Dao
|
ef0ed10622
Add window_size option to MHA and GPT
|
vor 11 Monaten |
 jiaxingli
|
386e391117
Fix: implement deterministic backward in mha (#748)
|
vor 1 Jahr |
|
Tri Dao
|
3f7d5786ba
Pass alibi slopes to flash_attn_with_kvcache during generation
|
vor 1 Jahr |
|
Tri Dao
|
c3b2196652
Add Alibi to MHA, test with Baichuan-13B
|
vor 1 Jahr |
|
Tri Dao
|
0705d2718d
[Llama] Fix some tests, add tests for Llama 2 and CodeLlama
|
vor 1 Jahr |
|
Tri Dao
|
e6a8026489
[Gen] Rename max_sequence_len->max_seqlen, sequence_len_offset->seqlen_offset
|
vor 1 Jahr |
|
Tri Dao
|
dfe29f5e2b
[Gen] Don't use ft_attention, use flash_attn_with_kvcache instead
|
vor 1 Jahr |
|
Tri Dao
|
8a733cbd53
[Gen] Fix calling update_graph_cache in tests
|
vor 1 Jahr |
|
Tri Dao
|
a86442f0f3
[Gen] Use flash_attn_with_kvcache in generation
|
vor 1 Jahr |
|
Tri Dao
|
798858f9f1
Fix test_baichuan
|
vor 1 Jahr |
|
Tri Dao
|
de2949f37d
[Rotary] Pass max_seqlen from mha.py to rotary during inference
|
vor 1 Jahr |
 dan_the_3rd
|
011ec323d6
Support MQA + MP for decoding (#490)
|
vor 1 Jahr |
|
Tri Dao
|
a2974e850a
Change causal for CrossAttention in mha.py to align to bottom right
|
vor 1 Jahr |
|
Tri Dao
|
f1a73d0740
Run isort and black on python files
|
vor 1 Jahr |
 Xuechen Li
|
bb4cded17b
support when num_heads is not divisible by world_size; resolves #459 (#461)
|
vor 1 Jahr |
|
Tri Dao
|
bec5b3d374
[MHA] Run black on mha.py
|
vor 1 Jahr |
|
Tri Dao
|
425dbcb6c6
[MHA] Implement MQA/GQA
|
vor 1 Jahr |
|
Tri Dao
|
4f285b3547
FlashAttention-2 release
|
vor 1 Jahr |
|
Tri Dao
|
62e9814466
[Rotary] Make sure frequency calculation is in fp32
|
vor 1 Jahr |
|
Tri Dao
|
48bc6eacd6
[Gen] Add rotary base as an argument to FT attention kernel
|
vor 1 Jahr |
|
Tri Dao
|
ba2fe7f378
[Gen] Move allocate_inference_cache to within the model
|
vor 1 Jahr |
|
Tri Dao
|
311d6606bf
[Gen] Fix FT kernel smem size, CG when batch size changed
|
vor 1 Jahr |
|
Tri Dao
|
ac3b684cdb
Have a separate nn.Dropout module in SelfAttention module
|
vor 1 Jahr |
|
Tri Dao
|
605655bc66
[Gen] Fix FT kernel when using CG
|
vor 1 Jahr |
|
Tri Dao
|
f5d0fbd468
[FT] Fix FT's single query attention for bf16 hdim128 rotary
|
vor 1 Jahr |
|
Tri Dao
|
4d87e4d875
Implement GPT-J
|
vor 1 Jahr |
|
Tri Dao
|
780e8eeabb
[ViT] Support timm checkpoint, add tests
|
vor 2 Jahren |
