DeepSpeed

Microsoft’s open-source deep learning optimization library for efficient distributed training and inference. DeepSpeed’s ZeRO (Zero Redundancy Optimizer) stages let you train models that exceed a single GPU’s VRAM by sharding optimizer states, gradients, and parameters across multiple GPUs. For multi-GPU AI workstations, DeepSpeed is the go-to tool for training models that don’t fit on one card.