 Tri Dao
|
5ca83a9c71
Clean up softcapping bwd a bit
|
5 months ago |
|
Tri Dao
|
751c762c9c
Don't specialize for hdim 224 to speed up compilation
|
5 months ago |
 Driss Guessous
|
1c275eb070
Fix ima for split-kv kernel (#1085)
|
5 months ago |
 janEbert
|
3c4053b75c
Make FA3 externally importable (#1053)
|
5 months ago |
 rocking
|
d8f104e97a
Support AMD ROCm on FlashAttention 2 (#1010)
|
5 months ago |
 Ying Zhang
|
dfe1a59e4b
Add var-seq-len to FA3 fp16 / bf16 fwd (#1072)
|
5 months ago |
 Cameron Shinn
|
cb516f855b
Remove torchlib dependency from cpp files (#1083)
|
5 months ago |
 Phil Wang
|
5f1ae4a34b
backwards for softcapping (#1033)
|
5 months ago |
 youkaichao
|
ef3e358a25
remove lambda (#1056)
|
5 months ago |
 Jorge António
|
4df62e1440
catch typo (#1058)
|
5 months ago |
|
Tri Dao
|
74b0761ff7
[FA3] BF16 forward
|
5 months ago |
|
Tri Dao
|
898dd4bbf2
Pass seqused_k to _flash_attn_varlen_forward
|
5 months ago |
|
Tri Dao
|
7ef24848cf
Add FA3 image
|
5 months ago |
|
Tri Dao
|
7f67966cc7
FA3 initial code release
|
5 months ago |
|
Tri Dao
|
b4a9dd6c9c
Temporarily switch to cutlass fork for more shapes
|
5 months ago |
|
Tri Dao
|
7551202cb2
Bump to v2.6.1
|
5 months ago |
|
Tri Dao
|
844912dca0
[CI] Switch from CUDA 12.2 to 12.3
|
5 months ago |
|
Tri Dao
|
40e534a7f6
Implement cache_leftpad
|
5 months ago |
|
Tri Dao
|
116b05f9b0
[CI] Compile with pytorch 2.4.0.dev20240514
|
6 months ago |
|
Tri Dao
|
da11d1b853
Bump v2.6.0
|
6 months ago |
|
Tri Dao
|
d0787acc16
Relax dropout_fraction test
|
6 months ago |
|
Tri Dao
|
dca6d89da4
Don't support softcap and dropout at the same time
|
6 months ago |
|
Tri Dao
|
81e01efd4b
More typo fixes
|
6 months ago |
|
Tri Dao
|
72e27c6320
Fix typo with softcapping
|
6 months ago |
|
Tri Dao
|
3d41db3e2c
Only test backward if there's no softcapping
|
6 months ago |
|
Tri Dao
|
908511b2b6
Split into more .cu files to speed up compilation
|
6 months ago |
|
Tri Dao
|
1d536d7de5
Minor cleanup of softcapping
|
6 months ago |
|
Tri Dao
|
beb2bf2a32
Drop support for pytorch 1.12, 1.13, and python 3.7
|
6 months ago |
|
Phil Wang
|
f4628b43ec
missing commas and backwards return arguments (#1032)
|
6 months ago |
 Nicolas Patry
|
8f873cc6ac
Implement softcapping. (#1025)
|
6 months ago |
