00Abstract
Long-context training runs on sequences whose lengths span orders of magnitude, and dynamic context parallelism (DCP) gives each sequence its own CP degree. Existing DCP systems either do not scale or perform poorly on mainstream models, leaving Megatron-Core (Mcore) DCP as the only option at production scale. Mcore DCP, however, sizes each degree to fit memory, which grows linearly with length while attention grows quadratically, so comparable token counts hide unequal computation. In our production 256K-context training job on more than 11K GPUs under Mcore DCP, per-rank microbatch times differ by up to 5× at similar token counts. The skew leads to a 46% pipeline bubble and a 13% data-parallel bubble.
We present HyDra, a scalable load-driven DCP system. Its scheduler balances computation by pulling every rank toward one load target, and balance in turn makes that target solvable in closed form. It places sequences with lazy heaps rather than whole-pool scans. That balance asks for CP degrees larger than memory requires, so its CP engine nests an inner Ulysses group in a shallow outer ring, letting a higher degree lower computation and communication together.
Evaluation at both scales shows consistent gains. On a 512-GPU testbed, HyDra raises throughput over Mcore DCP by 1.18× on average at 32K context and 2.48× at 256K. On a 2,048-GPU production job, it shrinks the pipeline bubble from 36% to 14%, cuts scheduling time by 2.3×, and raises throughput by 1.10–1.43× (avg. 1.25×) over Mcore DCP and 1.33–1.90× (avg. 1.59×) over static CP.
01Comparable tokens, unequal computation
Real long-context corpora mix many short documents with a few very long ones: in our 256K dataset, more than 85% of sequences are below 2K tokens and fewer than 0.6% exceed 32K. DCP handles this by giving each sequence its own CP degree, cut from a shared hybrid data-parallel (HDP) pool of DP×CP ranks.
The DCP scheduler shipped in Megatron-Core sizes each degree from a per-rank token budget, so every sequence goes to the fewest ranks that hold it. The budget counts tokens, but attention work grows as \(s^2\). We traced Mcore DCP on a 256K-context job with 11K+ GPUs (Hy4-preview, 770B MoE, \(tp{=}2, pp{=}9, ep{=}32\)) and found four problems.
Per-rank tokens are balanced. Attention is not.
Across the microbatches (MBs) of one iteration, per-rank token counts stay of the same order (about 5K on average), while per-rank attention intensity \(\mathcal{I}=\sum_i s_i^2/R\) spans four orders of magnitude, from about 0.1M to about 1,000M.
The heaviest MB runs more than 5× longer than the lightest. Every stage must process it, so it sets the critical path and widens the PP bubble from a ~10% balanced floor to 45.97%.
Insight 1. MBs must be balanced by attention computation, not by tokens alone.
Imbalance inside an MB accumulates into a DP bubble.
Within one MB, the busiest CP group carries up to 2.4× the computation of the idlest. Every MoE dispatch all-to-all is a barrier, so the slowest CP group's cost is imposed on its whole 32-rank EP group: one rank's attention takes 70 ms while the others finish in ~12 ms and wait ~59 ms.
DP replicas do not synchronize again until gradient reduction, so the skew grows from MB to MB and shows up as an 8–12.8% DP bubble. The slowest replica's gradient reduce-scatter runs up to 6.2× as long as the fastest.
Insight 2. Balancing must also cover CP groups within each MB and DP replicas across the iteration.
The scheduler itself idles GPUs for 8–9% of each iteration.
Mcore DCP plans on the CPU at the start of each iteration's data-fetch path, so the plan is on the critical path. Picking a degree is cheap. The cost comes from building the plan incrementally: for each sub-sample, the scheduler linearly scans the whole HDP pool to choose ranks, rebalance, and fill idle ranks.
Long-context training makes both factors large at once: a batch holds tens of thousands of short sub-samples, and the pool spans tens of thousands of ranks. A cost that is negligible at small scale becomes the orange scheduling band that opens every iteration.
Insight 3. Online planning must keep placement quality without sweeping the whole pool for every sequence.
Balance asks for a larger CP degree, but native CP cannot run it.
The MB that paces the iteration is attention-bound: on its bottleneck rank (one 256K sequence at memory-driven \(cp{=}32\)), FlashAttention takes 62.4% of a layer. Only a larger degree lowers \(s^2/cp\).
- Ulysses cuts both compute and traffic as \(1/P\), but it stops at the head count (\(cp \le 32\) at \(tp{=}2\)), and its QKV/output all-to-alls are exposed on the critical path (~4% of each iteration).
- Ring reaches any degree, but its per-rank traffic \(\approx 2\,s\,h_{kv}\) never shrinks. Under MLA, where \(h_{kv}\!\approx\!h\), it already exceeds attention compute.
Insight 4. A CP scheme must scale past the head count without making communication the bottleneck.
02HyDra: load-driven DCP
HyDra turns the four insights into four principles and realizes them with two co-designed components inside Megatron-LM (about 3K lines of Python).
- Balance computation, not just tokens, while keeping the memory feasibility DCP already provides.
- Balance within and across MBs: CP groups inside an MB and DP replicas across the iteration.
- Keep online planning lightweight: no whole-pool work as clusters grow.
- Scale CP without a communication bottleneck: degrees beyond the head count, with traffic that shrinks as the degree grows.
Neither component works alone: native CP cannot reach the target the scheduler sets, and the degrees the engine supports pay off only when a balanced plan requests them.
One load target, in closed form
Even at the cap \(C\), the longest sequence leaves \(s_{\max}^2/C\) on each of its ranks. This is the lowest feasible target, and also the easiest for token-limited MBs of short sequences to reach.
Iteration time with skew is a graph of waits with no closed form. Under balance, a larger cap trades a shorter pipeline bubble (\(\beta/C\)) against more MBs, each with a fixed cost (\(\gamma C\)). Only the ratio \(\theta/c\) needs to be calibrated.
The smallest power-of-two degree that meets both the computation target and the token budget \(B\). Mcore enforces only \(s/B\).
Try it: memory-driven vs. load-driven CP degrees
Eq. (9) in the 256K production setting (\(s_{\max}=256\text{K}\), \(B=8\text{K}\), so \(C_{\text{mem}}=32\)). Drag the cap \(C\) to see how HyDra gives long sequences more ranks and flattens the heaviest per-rank load.
Table view
Illustrative, per-sequence view of the paper's equations. Short sequences are not left underloaded: piggyback packing fills ranks with them up to \(\Lambda^\star\) or the token budget. Packing, communication, and pipeline effects are not modeled here.
Piggyback packing
Sequences are placed longest-first, one MB at a time across all \(G\) ranks. A sequence needing degree \(k\) opens a new \(k\)-aligned group if enough ranks are free. Otherwise it piggybacks on the smallest feasible open group of size \(\ge k\), with ties broken by load. Short sequences fill the gaps next to wide long-sequence groups, so fewer ranks sit idle and fewer MBs are needed. Because every MB is packed against the same \(\Lambda^\star\), the balance extends to the CP and EP groups inside an MB and to the DP replicas after it.
Heap-accelerated scheduling
Open groups are partitioned by CP degree. Each class keeps a lazy min-heap keyed by load and a max-heap keyed by remaining token capacity. Stale entries are dropped on access, and a candidate that violates the budget is demoted instead of rescanned. Classes whose minimum already reaches \(\Lambda^\star\) are pruned. The result is the same least-loaded placement as a linear search, with the same tie-breaking.
Balance-preserving CP communication
Inner Ulysses × outer ring
Up to the head count \(h\), the degree is pure Ulysses. Beyond it, a shallow outer ring supplies the remaining factor, so the scheduler may request any \(P\le C^\star\) while per-rank compute stays \(s^2/P\). The ring's flat traffic floor is confined to the residual factor: at \(P{=}256\) and \(h{=}64\), the outer ring spans only four ranks.
Exposed all-to-all overlap
Rather than materializing Q, K, and V and then issuing three blocking collectives, HyDra launches each all-to-all on a separate stream as soon as its tensor is final, and hides it behind the independent branches of gated MLA. Q goes first (covered by the KV projections), then V (covered by K's RoPE and concat), then K (covered by the gate projection). Launch and wait are paired autograd functions, so backward gets the mirrored overlap automatically.
03Results
We evaluate at three scales: a 512-GPU NVIDIA H20 testbed, a production job on 2,048 NVIDIA H800 GPUs (both on an internal MLA-based MoE model with \(tp1, pp4, ep8\)), and simulation at 8K–40K GPUs. Baselines on hardware are static CP and Mcore DCP. Simulation adds ByteScale, WLB-LLM, and FlexSP.
512-GPU testbed
One iteration across the four PP stages at 32K context (same time axis).
One iteration across the four PP stages at 256K context (same time axis). Under Mcore DCP the stages are full of gaps. Under HyDra they stay packed and the iteration ends much earlier.
2,048-GPU production job · 256K context
One iteration across the four PP stages on 2,048 GPUs (same time axis).
Where the CP degrees go
HyDra spends context parallelism where it balances MBs. Mcore DCP never exceeds the memory-driven \(cp{=}32\). HyDra puts 45.5% of GPUs at \(cp \ge 64\) and 22.7% at \(cp{=}256\), which the inner-Ulysses × outer-ring engine makes affordable.
What it saves
For an illustrative 2T-token budget at these rates, training takes 17.4 days under static CP, 13.7 under Mcore DCP, and 11.0 under HyDra. At $2 per GPU-hour that is $632K saved over static CP and $271K over Mcore DCP on 2,048 GPUs.
Scheduling overhead
Scheduling runs on the CPU, so we can measure it at any rank count. Mcore DCP's whole-pool scans reach 47.85 s at 16K GPUs, while HyDra takes 4.12 s, an 11.6× speedup. On the 2K-GPU production run, online scheduling drops from 0.78 s to 0.34 s (2.3×).
Model convergence
HyDra reorders gradient and loss reductions, so small floating-point differences are expected. Over roughly 8,000 iterations its language-modeling loss stays within 10−3 of static CP, and it records the lower loss in about half of the iterations: unbiased noise rather than drift.
Scale-out simulation, 8K–40K GPUs
An in-house simulator replays batches through the full parallel schedule with operator costs measured on the same hardware. Validated against per-rank production traces at 11K+ GPUs, it lands within 10% of the measured mean iteration time.
On Hy3 (GQA) at 8K GPUs, HyDra improves over the strongest baseline, ByteScale, by 1.28×, over Mcore DCP by 1.53×, over WLB-LLM by 1.56×, and over FlexSP and static CP by 1.7–2.2×. It is not the best on any single axis. It wins because it keeps the PP bubble (19.8%) and DP bubble (3.6%) low together without paying static CP's communication cost.
Under weak scaling to 40K GPUs, HyDra is fastest at every scale. Its lead is wider on the MLA model DeepSeek-V3: 1.53× over ByteScale, 1.61× over Mcore DCP, and up to 13.8× over ring static CP.
04Paper & citation
@misc{fan2026hydra,
title = {HyDra: Demystifying and Taming Dynamic Context
Parallelism at Production Scale},
author = {Fan, Zihao and Liu, Yunzhuo and Jiang, Bo and
Zheng, Changgang and Zheng, Lin and Ying, Ray and
Zhang, Key},
year = {2026},
note = {Preprint}
}