Distributed Training

Training

• Thread on dataloaders

• Picotron (nanoGPT for 4D parallelism)

  • Picotron tutorial Part 1: Model, Process Group Manager, Dataloader

• The Ultra-Scale Playbook

• Megatron blogpost (scatter-gather optimization, Performance microbenchmarks for pipeline parallelism, )

• GSPMD: General and Scalable Parallelization for ML Computation Graphs

  • GSPMD is now the fundamental component of JAX/TensorFlow distributed training and enables various optimizations with the XLA compiler to allow users to train their models efficiently in a large scale setting.

• Visualizing 6D Mesh Parallelism

• Universal Checkpointing https://x.com/stasbekman/status/1808287880781127930?s=12