# GB200 NVL72 Checkpoints: When Async S3 Tiering Wins on Cost

Wei Chen · September 1, 2026

> GB200 NVL72 Checkpoints: When Async S3 Tiering Wins on Cost. A GB200 NVL72 rack holds 13.5 TB of HBM3e across 72 Blackwell GPUs and c...

| Takeaway | Detail |
| --- | --- |
| Checkpoint I/O is a burst workload, not sustained throughput | A 3.2 TB optimizer checkpoint completes in a 90-second I/O burst, leaving storage idle for the remaining 59 minutes of an hourly training loop |
| Parallel filesystems overpay for unused capacity | Paying 10x per GB-month for parallel FS throughput you use 1% of the time creates a cost inefficiency comparable to renting a datacenter for a cron job |
| Metadata contention limits checkpoint performance more than raw bandwidth | Large multipart objects or aggregated files reduce metadata pressure during checkpoint operations, preventing stalls that throttle GPU productivity |
| Compression directly reduces storage and network costs | Applying nvCOMP or TurboQuant compression techniques optimizes data transfer speeds and significantly lowers storage costs for large-scale AI factories |

A GB200 NVL72 rack holds 13.5 TB of HBM3e across 72 Blackwell GPUs and can emit a full 3.2 TB optimizer checkpoint as a 90-second I/O burst, then touch storage zero times for the next 59 minutes. This arithmetic exposes a fundamental flaw in legacy AI storage planning: treating model checkpoints like traditional HPC data forces teams onto Lustre or GPFS architectures that demand sustained throughput they will never actually consume.

The contrarian claim that AI checkpoints belong on high-performance parallel filesystems collapses at GB200 scale-and-cadence. Checkpoint writes must complete within a tolerable stall window measured in seconds to low tens of seconds to avoid stalling GPUs, making burst write bandwidth the true requirement rather than continuous IOPS. When you price both storage paths for that exact shape, the parallel filesystem loses before a single byte is benchmarked.

Paying 10x per GB-month for parallel FS throughput you use 1% of the time is the storage equivalent of renting a datacenter for a cron job. Async S3 tiering aligns with the actual workload profile by absorbing bursts without throttling, while metadata contention becomes the limiting factor only when poorly structured. Aligning checkpoint writes to zone boundaries or using aggregated multipart objects further reduces garbage collection overhead, proving that object storage is the economically rational default for modern training loops.

![GB200 NVL72 Checkpoints](https://static.mm-ais.com/article-images-ai/gb200-nvl72-checkpoints-when-async-s3-ti-ai-e1fbc103.jpg)

## From HBM to HTTPS

The checkpoint I/O shape on a GB200 NVL72 is defined by massive, discrete bursts rather than sustained streaming. Each node's 72 Blackwell GPUs hold 13.5 TB of HBM3e, and a distributed checkpointer executing a PyTorch DCP or Megatron-style sharded save emits hundreds of gigabytes of BF16 model weights alongside roughly 2.4 TB of FP32 AdamW optimizer state. This totals ~3.2 TB emitted as a single burst per save event. According to OneSource Cloud (2026-08-01), checkpointing during training demands burst write bandwidth rather than sustained throughput, and storage must absorb these bursts without throttling to maintain GPU productivity. The critical constraint is that checkpoint writes must complete within a tolerable stall window measured in seconds to low tens of seconds to avoid stalling the GPUs, which forces the data path to decouple the GPU compute loop from the persistent storage write.

S3 tiering becomes viable through a three-stage data path that isolates the GPU stall from the network egress. First, sharded checkpoint slices stream from HBM over NVLink/NVSwitch into each node's local NVMe; this absorbs the burst instantly and decouples the GPU stall from the slower network write. Second, a background uploader on each node performs S3 multipart uploads using 128 MB parts with high per-node concurrency. Third, training resumes the moment NVMe absorbs the shard, meaning the S3 leg never blocks the step loop. This architecture converts the collective rendezvous required by parallel filesystems into independent per-node operations. According to disks.us (2026-01-04), metadata contention is often the limiting factor for checkpoint performance, not raw device bandwidth. By staging to NVMe first, the system avoids the MDS bottleneck inherent in parallel FS paths where 72 ranks simultaneously open tens of thousands of shard files. Furthermore, using large multipart objects reduces metadata pressure during checkpoint operations, as noted by disks.us (2026-01-04).

| Component | Metric / Behavior | Impact on Checkpoint Path |
| --- | --- | --- |
| S3 Express One Zone | Single-digit-millisecond latency; high per-stream PUT bandwidth | Per-node upload speed matches NVMe drain rate; no network bottleneck at ingest tier. |
| Concurrent Uploaders | Throughput scales linearly with streams | 64 NVL72 nodes uploading in parallel yield an aggregate ceiling of ~70 GB/s. |
| Checkpoint Drain Time | 3.2 TB drained at ~70 GB/s aggregate | Saturates the rack network before saturating S3; completes faster than most training loops can quiesce. |
| FSx for Lustre | Published ~1 TB/s aggregate capacity | Requires provisioned, mounted, and warm 24/7 across all nodes; idle utilization drops to ~2.5% for hourly saves. |
| MDS / Metadata Server | Contention point for 72 ranks opening tens of thousands of shards | Creates rendezvous stalls if a single rank lags or directory creation hot-spots occur. |
| NVMe Staging | Converts collective stall to independent uploads with retry | Avoids failure modes where one slow rank blocks the entire cluster's save operation. |

The S3-side concurrency model eliminates the myth that object storage is too slow for AI checkpoint I/O. A single EC2 instance sustainably pushes high per-stream PUT bandwidth to S3 Express One Zone. When scaled across 64 concurrent NVL72 nodes in one Availability Zone, the aggregate ceiling reaches ~70 GB/s. This means the 3.2 TB checkpoint drains the network before it saturates S3, completing the transfer in under a minute. This throughput is sufficient to handle the burst without blocking the training loop, directly contradicting the assumption that object storage cannot keep pace with GPU-scale checkpoint rates. The idle math further justifies this approach: one checkpoint per hour at 90 seconds of burst means parallel FS capacity is utilized only ~2.5% of wall-clock time. In contrast, S3 object storage costs accrue strictly for bytes stored and PUTs issued, avoiding the fixed cost of maintaining a warm, provisioned filesystem that sits idle for 59 minutes and 30 seconds between saves.

![From HBM to HTTPS — GB200 NVL72 Checkpoints](https://static.mm-ais.com/article-images-ai/gb200-nvl72-checkpoints-when-async-s3-ti-ai-848df7cc.jpg)

## The Evidence

The performance and cost gap between object storage tiering and parallel filesystems for GB200 NVL72 checkpointing is not theoretical; it is defined by the specific throughput ceilings of S3 Express One Zone, the prefix scaling limits of S3 Standard, and the fixed capacity pricing of FSx for Lustre. The myth that object storage cannot handle AI checkpoint I/O collapses when you examine the per-stream bandwidth available on high-bandwidth EC2 instances. According to AWS 'Amazon S3 Express One Zone' performance documentation, S3 Express One Zone supports hundreds of thousands of requests per second and delivers up to 6x faster first-byte latency than S3 Standard, with per-stream PUT throughput on high-bandwidth instances that rivals local NVMe write speeds. This means a single node can sustain a full-disk drain rate, and 64 concurrent nodes in one AZ can drain a 3.2 TB state in under a minute—faster than most training loops can quiesce.

However, raw throughput is only half the equation; the multipart upload architecture must respect S3's internal scaling limits. According to AWS S3 performance guidelines, each S3 Standard prefix is subject to per-prefix request-rate limits on PUT/COPY/POST/DELETE and GET/HEAD operations. A 3.2 TB checkpoint split into standard part sizes will generate enough metadata operations to hit those limits and self-throttle if partitioned poorly. This per-prefix request ceiling is the hard constraint that dictates your part size and worker count: you must either use multi-prefix partitioning to linearize request rates or ingest directly into S3 Express One Zone, which removes the prefix bottleneck entirely. The decision here is binary: if you stay on S3 Standard, your part size and worker concurrency are capped by this prefix scaling rule; if you use Express, you bypass it.

The checkpoint size itself drives the urgency of these decisions. According to Meta's Llama 3 herd-of-models paper, the largest model in the herd trained in BF16 yields hundreds of gigabytes of weights; multiplying by ~4x for AdamW FP32 master weights and momentum/variance buffers results in the ~3.2 TB full-state checkpoint used throughout this analysis. This volume confirms that the workload is a discrete, massive burst rather than sustained streaming. For such bursts, asynchronous tiering to S3 Express One Zone followed by a lifecycle policy to S3 Standard captures the best of both worlds: high-speed ingest without throttling, and archive-grade durability at low cost. According to AWS S3 SLA and durability documentation, S3 Standard offers 99.999999999% durability while Express One Zone provides a 99.95% availability SLA, meaning tiered checkpoints get 11-nines durability without a separate backup pipeline that a parallel filesystem would still require. The evidence converges on a single conclusion: use S3 Express One Zone as the ingest tier for hourly-or-slower checkpointing, and reserve parallel filesystems only for frequencies exceeding every ~15 minutes or where POSIX sharing is mandatory.

| Metric | S3 Express One Zone (Ingest) | S3 Standard (Lifecycle Target) | FSx for Lustre Persistent SSD |
| --- | --- | --- | --- |
| Per-Stream PUT Throughput | High per-stream PUT bandwidth (high-bandwidth instances) | N/A (Read-only target) | Fixed ratio per TiB provisioned |
| Prefix Scaling Limit | Hundreds of thousands RPS (no prefix bottleneck) | Per-prefix PUT/GET request-rate limits | N/A (POSIX namespace) |
| Aggregate Throughput Scale | Linear with instance count | N/A | ~1 TB/s at large provisioned sizes |
| Storage Cost (us-east-1) | Premium ingest rate (tiered out after 24h) | Low per-GB-month storage rate | Flat billing for provisioned capacity regardless of usage |
| Request Cost Model | Low-latency ingest optimized | Per-request pricing for PUTs | Flat capacity bill (no request fees) |
| Durability / Availability | 99.95% availability SLA | 99.999999999% (11 nines) durability | Dependent on replication config |

At a checkpoint frequency of one per hour for a 3.2 TB model state, the I/O shape on a GB200 NVL72 rack resolves to a discrete burst lasting roughly two to four minutes. This duration is dictated by the time required to drain local NVMe and quiesce the training loop, not by network saturation. Any storage tier capable of sustaining an aggregate write bandwidth exceeding 15 GB/s clears this window without blocking the GPU compute path. Once that threshold is crossed, aggregate bandwidth ceases to be the discriminator; the decision variable shifts entirely to cost-per-idle-hour and operational friction. In this regime, object storage wins by an order of magnitude because it charges for capacity only when data exists, whereas parallel filesystems incur a flat bill for provisioned throughput regardless of whether the cluster is training or idling.

![The Evidence — GB200 NVL72 Checkpoints](https://static.mm-ais.com/article-images-pixabay/gb200-nvl72-checkpoints-when-async-s3-ti-47e4eba7.jpg)

## The Frequency Fork

The myth that object storage is too slow for AI checkpoint I/O collapses under the mechanics of S3 Express One Zone. A single EC2 instance can sustainably push high per-stream PUT bandwidth to S3 Express One Zone. With 64 concurrent nodes in a single Availability Zone, the aggregate ingest capacity drains 3.2 TB in under a minute. This throughput exceeds the rate at which most training loops can even quiesce, proving that async multipart uploads from NVMe into S3 Express are not a bottleneck but a decoupling mechanism that eliminates contention during the critical save phase.

For the thesis regime of checkpoints occurring hourly or slower, S3 Express One Zone configured as an ingest tier with a 24-hour lifecycle transition to S3 Standard emerges as the dominant architecture. It wins every row that matters in this frequency band: cost efficiency, idle cost reduction, durability, and operational simplicity. The parallel filesystem options—whether FSx for Lustre or self-managed GPFS/WEKA on NVMe—win only the POSIX semantics and resumption-latency rows. However, these advantages are irrelevant if the workload does not require shared POSIX access between training phases. The tiebreaker column reveals the only legitimate reason to abandon the object storage tiering pattern: if your workflow demands mid-run POSIX dependencies, such as activation scripts or shared scratch space accessed concurrently by running jobs, S3 tiers lose by construction. In those specific cases, the runner-up filesystem becomes the winner, but this is a functional constraint, not a performance necessity.

| Dimension | S3 Standard (Direct) | S3 Express One Zone (Tiered Ingest) | FSx for Lustre (Persistent SSD) | Self-Managed GPFS/WEKA on NVMe |
| --- | --- | --- | --- | --- |
| Aggregate Write Bandwidth | Limited by prefix GET limits; requires shard fan-out design | High per-stream PUT bandwidth; clears 3.2 TB bursts rapidly | High; provisioned throughput scales with file system size | Very High; limited only by host PCIe/NVMe topology |
| Per-GB-Month Cost | Lowest tier pricing; lifecycle transition reduces average cost | Premium ingest-tier pricing before lifecycle transition | Premium for persistent SSD class; flat provisioning cost | Hardware depreciation + management overhead; no cloud premium |
| Idle Cost (No Checkpoint Flight) | Negligible; pay only for stored bytes after lifecycle moves to Standard | Minimal after 24-hour lifecycle transition to Standard | High; you pay for provisioned capacity whether used or not | High; hardware sits idle consuming power and capital |
| POSIX Semantics | None; API-based access only | None; API-based access only | Full POSIX compliance; shared scratch between phases | Full POSIX compliance; native shared scratch support |
| Resumption Read Latency | Budget 3–5 minutes for full restore; design shard fan-out to mitigate prefix limits | ~1 minute at full rack concurrency; reads match ingest speed | Sub-minute; direct block access with low tail latency | Sub-minute; lowest possible latency via local NVMe |
| Ops Burden | Low; managed service with lifecycle policies | Low; managed service; async upload simplifies client logic | Medium-High; tuning Lustre MDS/OSS; patching | High; self-managed cluster ops, hardware failures, tuning |
| Tiebreaker: Mid-Run POSIX Dependencies | Loss; cannot serve activation/replay scripts mid-run | Loss; cannot serve activation/replay scripts mid-run | Win; supports shared scratch and POSIX semantics | Win; supports shared scratch and POSIX semantics |

The crossover condition where parallel filesystems regain superiority occurs when checkpoint frequency exceeds roughly four per hour, or an interval of approximately 15 minutes. At this density, the economics invert. The per-PUT overhead and the higher storage pricing of S3 Express One Zone begin to rival the flat bill of a warm filesystem. More critically, the async upload pipeline can no longer drain the local NVMe fast enough between saves. If the interval shrinks below the time required to complete the multipart upload cycle, the asynchronous advantage evaporates, and the synchronous latency of the filesystem becomes preferable. For runs checkpointing every 15 minutes or faster, provision a parallel filesystem. For everything slower, stage to NVMe, async-upload to S3 Express, and let the lifecycle policy handle the rest.

The cost and throughput advantages of async tiering hold only when the workload shape matches the storage architecture. When teams operate preemption-heavy training on Spot or interruptible GB200 capacity, the checkpoint cadence often shifts to every 5–15 minutes to limit lost work to minutes rather than hours. At that frequency, the asynchronous uploader cannot drain 3.2 TB of local NVMe between saves; the burst duration exceeds the interval, causing I/O backpressure that stalls the training loop. In this regime, the parallel filesystem's predictable 1 TB/s write bandwidth genuinely wins because it handles the high-frequency bursts without queuing artifacts in a background pipeline. According to disks.us (2026-01-04), parallel filesystems deliver consistent throughput but require heavy metadata coordination, making them better suited for massive parallel training with tightly-coupled workloads where checkpoint latency must remain deterministic regardless of network conditions.

![The Frequency Fork — GB200 NVL72 Checkpoints](https://static.mm-ais.com/article-images-pixabay/gb200-nvl72-checkpoints-when-async-s3-ti-e94bc667.jpg)

## What the Data Doesn't Tell You

Vendor benchmarks for S3 Express One Zone report high per-stream PUT throughput, but these are ceilings measured under ideal isolation. Real training clusters share NICs with gradient all-reduce traffic, and observed upload rates can fall 30–50% during synchronized save windows when NCCL is simultaneously saturating EFA or VPC networking. The async uploader competes for the same physical links used to synchronize model weights across the rack. If the checkpointer does not throttle uploads relative to NCCL congestion signals, the combined load can push packet loss above acceptable thresholds, triggering retransmissions that further degrade both training convergence speed and checkpoint completion time. Platform teams must instrument end-to-end metrics that account for shared transport contention, not just isolated object-store performance.

Benchmarks also omit the API-semantics gap that breaks many existing codebases. Older DeepSpeed and Megatron save hooks, along with certain tokenizers and dataloader caches, rely on shared POSIX filesystems for rendezvous files or atomic renames to coordinate distributed saves. S3 has no rename operation; porting requires refactoring checkpointer logic to use write-then-tag or manifest-object patterns. This migration cost—engineering hours, regression testing, and potential runtime overhead—is never reflected in the price comparison table. According to Digital Thought Disruption (2026-07-24), PowerScale NFS remains recommended when many workers or tenants need concurrent POSIX-style access to shared datasets and distributed checkpoints, precisely because it preserves the atomic file operations that legacy frameworks expect. Teams evaluating async tiering must audit their checkpoint stack for POSIX dependencies before assuming a drop-in replacement is viable.

The cited 10x storage cost gap carries significant uncertainty bands that writers often flatten into a single ratio. The calculation ignores request charges at very high part counts: a badly tuned 64 MB part size on a 3.2 TB checkpoint yields a very large number of PUT requests per save, which accumulates meaningful costs at scale. It also excludes egress fees if readers live outside the Express AZ and lifecycle-transition costs as objects move from Express to Standard. A prudent cost model should present a wide uncertainty band around the headline ratio to account for these variables. Furthermore, figures here are AWS-specific; CoreWeave, OCI, and on-prem object stores like MinIO or Ceph RGW have materially different per-stream PUT ceilings and none publish S3-Express-class single-digit-ms guarantees. Consequently, the crossover threshold of roughly four checkpoints per hour shifts depending on the object store chosen, and the async tiering advantage may vanish entirely on platforms with higher request pricing or lower concurrency limits.

Finally, durability numbers hide a silent-failure risk that parallel filesystems avoid by design. An async tiering pipeline can fail in the background due to an uploader crash, credential expiry, or throttled prefix, yet the training run proceeds for hours believing checkpoints exist. If the failure goes undetected until the next scheduled inspection, the team loses the entire window of progress. Parallel filesystem writes are synchronous and fail loudly; if the disk fills or metadata locks up, the checkpointer aborts immediately, alerting operators to the problem. This operational argument for FS is real and often decisive for production reliability, even when the cost analysis favors object storage. According to OneSource Cloud, AI storage architecture must simultaneously serve training throughput, checkpoint writes, inference data feeds, and model serving, meaning the storage choice must balance cost against the risk of silent state loss.

One GB200 NVL72 rack training a large Llama-3.1-class model in BF16 with AdamW generates a precise, predictable I/O signature: each checkpoint comprises hundreds of gigabytes of weights and ~2.4 TB of optimizer state, totaling 3.2 TB per snapshot. Over a standard 30-day run with hourly checkpoints, the system produces a discrete artifact every hour. The critical constraint is not aggregate volume but the quiesce window; the training loop must pause while this data moves off-chip. A common myth persists that object storage cannot drain a Blackwell rack fast enough to avoid stalling the compute pipeline. This belief ignores the sustained PUT bandwidth available on S3 Express One Zone. According to infrastructure benchmarks for EC2 instances in supported Availability Zones, a single node can sustainably push high per-stream PUT bandwidth to S3 Express. In a 72-node rack, shards are distributed such that each node stages only its local portion (~45 GB). At that per-stream rate, a node clears its 45 GB shard in roughly 41 seconds. With 72 nodes uploading concurrently within the same AZ, the entire 3.2 TB rack checkpoint becomes durable in under two minutes. This duration falls well within the acceptable quiesce tolerance for most large-model training loops, fully removing checkpointing from the critical path without requiring parallel filesystem overhead.

| Condition | Async Tiering Outcome | Parallel FS Outcome | Winner |
| --- | --- | --- | --- |
| Cadence > 4 checkpoints/hour (e.g., 5–15 min intervals) | NVMe backlog grows; I/O stalls training loop | Predictable 1 TB/s drains bursts instantly | Parallel FS |
| Legacy checkpointer requires POSIX atomic renames | Requires code refactor (write-then-tag); migration cost unquantified | Native support; zero code changes | Parallel FS |
| Spot/preemption-heavy runs tolerating

Canonical: https://x-oss.com/blog/gb200-nvl72-checkpoints-when-async-s3-tiering-wins-on-cost.php
Markdown: https://x-oss.com/blog/gb200-nvl72-checkpoints-when-async-s3-tiering-wins-on-cost.php/index.md