Installs into .claude/skills of the current project.
Are you the author of Distributed Training?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/fourteen1416-distributed-training)
---
name: distributed-training
description: Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs.
metadata:
category: tooling
trigger-keywords: "distributed,multi-gpu,parallel,ddp,scale"
applicable-stages: "10,12"
priority: "7"
version: "1.0"
author: researchclaw
references: "PyTorch DDP Tutorial, pytorch.org; Goyal et al., Accurate Large Minibatch SGD, 2017"
---
## Distributed Training Best Practice
1. Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
2. Initialize process group: dist.init_process_group(backend='nccl')
3. Use DistributedSampler for data sharding
4. Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
5. Only save checkpoint on rank 0
6. Scale learning rate linearly with world size
7. Use gradient accumulation for effectively larger batch sizes
