|
@@ -1,6 +1,9 @@
|
|
This CUDA extension implements fused dropout + residual + LayerNorm, based on
|
|
This CUDA extension implements fused dropout + residual + LayerNorm, based on
|
|
Apex's [FastLayerNorm](https://github.com/NVIDIA/apex/tree/master/apex/contrib/layer_norm).
|
|
Apex's [FastLayerNorm](https://github.com/NVIDIA/apex/tree/master/apex/contrib/layer_norm).
|
|
We add dropout and residual, and make it work for both pre-norm and post-norm architecture.
|
|
We add dropout and residual, and make it work for both pre-norm and post-norm architecture.
|
|
|
|
+
|
|
|
|
+It has only been tested on A100s.
|
|
|
|
+
|
|
```sh
|
|
```sh
|
|
cd csrc/layer_norm && pip install .
|
|
cd csrc/layer_norm && pip install .
|
|
```
|
|
```
|