Home Explore Help
Register Sign In
david
/
flash-attention
mirror of https://github.com/Dao-AILab/flash-attention
1
0
Fork 0
Files Issues 0 Wiki
Branch: main
Branches Tags
flash-attention
/
training
/
configs
/
experiment
/
pile
/
gpt3xl-flash-rotary.yaml

gpt3xl-flash-rotary.yaml 186 B
Permalink History Raw

12345678 
  1. # @package _global_
    2. 

  2. defaults:
    3. 

  3.   - /experiment/pile/gpt3xl-flash.yaml
    4. 

  4. 

  5. model:
    6. 

  6.   config:
    7. 

  7.     max_position_embeddings: 0  # Disable absolute position embedding
    8. 

  8.     rotary_emb_fraction: 0.5
    9.