NVIDIA 如何用自动化工作负载恢复保障分布式训练持续运行
Keep Distributed Training Running with Automated Workload Recovery
分布式 AI 训练规模扩大后，硬件与软件故障会中断训练进度，NVIDIA 提出自动化工作负载恢复来应对。一个训练 rank 停滞可能触发 NCCL 超时，GPU 节点可能不再响应调度器，慢节点则会拖慢整个任务。这些中断单独来看都可处理。
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery...
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own…
来源：NVIDIA Developer Blog · developer.nvidia.com