Fused optimizer

• Performance Comparison between Torch.Compile and APEX optimizers

  • TL;DR: Compiled Adam outperformed SOTA hand-optimized APEX optimizers on all benchmarks; 62.99% on Torchbench, 53.18% on HuggingFace, 142.75% on TIMM and 88.13% on BlueBerries

  • just torch.compile your optimizer step

• Implementing your own Adam step and torch.compile

  • https://x.com/cloneofsimo/status/1809169769977016501?lang=bn

• ms-amp fused optimizer

  • https://github.com/Azure/MS-AMP/blob/main/msamp/deepspeed/runtime/fp8/fused_optimizer.py