How Do You Benchmark Object Storage Without Measuring the Wrong Thing?

The Direct Answer: Benchmark the Workload, Not a Storage Product

A credible object storage benchmark measures the operations your application actually performs: object uploads, downloads, deletes, listings, multipart transfers, or reads through a database or analytics engine. Comparing vendor averages without specifying object size, request concurrency, client count, region, network path, and consistency behavior produces a marketing exercise rather than an engineering decision. The best test usually runs your own dataset and access pattern against at least two candidate services, using identical binaries, credentials, hardware, and test code wherever the services permit.

Also worth reading: How Do Platform Teams Master Cross-Cloud Storage Governance Without Breaking the Data Plane? · Which Multi-Cloud Storage Orchestration Strategies Work Best for Enterprise Object Storage in 2026? · What is the definitive guide to implementing object storage for startups in 2026?

For platform teams, the decisive measures are frequently p95 or p99 latency, sustained throughput, request-error rate, and total cost per completed workload. Average latency can hide unacceptable tail behavior, while aggregate throughput can conceal the small-object performance that damages interactive applications. A benchmark should therefore report a distribution and define whether failures, retries, queueing, and time spent provisioning infrastructure are included. Object storage is not a single product category with one performance score; it is a set of services, regions, access paths, and workload modes that must be tested under controlled conditions.

A useful baseline is 1 KiB, 1 MiB, 128 MiB, and 1 GiB objects, tested at 16, 64, and 256 concurrent clients. Many production estates are dominated by objects smaller than 1 MiB, while analytics, backups, and machine-learning pipelines often move 128 MiB or larger. Add a mixed-size workload with a documented distribution, because separate tests can make every result look better than a real pipeline in which metadata requests, large transfers, and indexing occur together.

What Makes Object Storage Performance Different?

Object storage performance depends on metadata operations, network round trips, consistency, and parallelism rather than only raw disk speed. A 1 MiB download may be limited by request overhead, connection setup, and service-side processing before bandwidth becomes the constraint. By contrast, a 1 GiB transfer is more likely to expose network capacity and multipart behavior. You should record operation latency separately for small and large objects instead of reporting one blended requests-per-second figure.

The path matters as well. A benchmark from one cloud region can be fast because compute, clients, and storage are colocated, while a cross-region application sees different latency and egress cost. Similarly, a public internet test does not model a private virtual network, an on-premises data center, or a cross-cloud connection. Record the client location, availability-zone placement, network interface, MTU, and whether TLS terminates on the client, proxy, or service. These details explain discrepancies that otherwise appear mysterious.

Consistency is another reason to validate behavior rather than assume compatibility. Services described as Amazon S3 compatible may differ in conditional writes, versioning, object locking, checksums, multipart limits, event notifications, and administrative APIs. Compatibility is valuable when ordinary GET and PUT requests behave as expected, but it is not proof that every feature is interchangeable. Include application-level verification, such as reading recently written objects and checking that overwrite and delete semantics remain correct under concurrency.

For database-backed tests, separate the object store from the database engine. ClickHouse benchmarking on a New York City taxi dataset can compare how MinIO and S3 affect query performance, but any observed difference may originate in ClickHouse settings, table design, server hardware, caching, or data placement. Keep those variables fixed, clear caches when appropriate, and publish the ClickHouse version and table schema. The object store is part of the system, but one number cannot fairly represent both products independently.

Build a Benchmark That Produces Reproducible Results

Begin by defining the workload before selecting tools. A backup benchmark might emphasize sequential multipart uploads, sustained throughput, and recovery; an image-processing pipeline may emphasize millions of small PUTs; an analytics cluster may emphasize repeated range reads and query completion time. Specify the dataset, access-key distribution, concurrency, duration, success criteria, and whether caching is enabled. A 30-minute warm-up followed by 60 minutes of measurement is often more informative than a single uncontrolled loop, although short tests can work for early screening.

Use maintained, documented drivers and record their versions. The S3 API has many optional parameters, and defaults can materially change behavior. A rigorous test should report checksum validation, multipart thresholds and part sizes, connection reuse, TCP window behavior, retry policy, and timeout values. Randomize test order when comparing services to reduce the effect of transient network conditions, and repeat each run at least three times. Report the median run plus the worst tail result rather than selecting the most favorable run.

Measure the client as carefully as the service. Capture CPU utilization, memory pressure, disk activity, network utilization, dropped packets, retransmissions, and per-process latency. A 10 Gbps client interface cannot validate a 100 Gbps claim, and 256 parallel clients can overload a single small virtual machine. Capacity planning should include the number of connections, sockets, open files, and temporary disk space required by your test harness. If the candidate platform provides its own benchmarking agent, inspect what that agent measures before treating its score as neutral evidence.

First, create equivalent buckets with versioning, encryption, lifecycle, and retention policies disabled unless those settings are explicitly part of the workload. Give each test identity only the required permissions, because administrative privileges should not affect GET or PUT performance. Use private networking where production will use it, and do not route one candidate through an internet gateway while the other uses a local endpoint. Generate a manifest of object keys and sizes in advance so provisioning differences do not distort the measured phase.

Next, run separate upload and download phases. For uploads, include single-part and multipart paths if production relies on both, and determine whether server-side copying is being measured. For downloads, use both sequential and random key order, because object stores may spread reads differently across nodes and regions. Measure small-object LIST operations independently from GETs. If the application searches buckets frequently, listing cost and latency deserve explicit treatment rather than being hidden in a general operations loop.

Then test the application. Load a known number of rows into a ClickHouse table backed by each store, run a fixed set of scans, joins, aggregations, and cache states, and record query duration plus object-store metrics. Public comparisons titled “MinIO vs. AWS S3” or “Cloudflare R2 vs. AWS S3” can provide hypotheses, but they are not substitutes for an internal test. Vendor results may use different hardware, data, concurrency, regions, or billing assumptions, and a 99% egress-saving claim says nothing by itself about request latency.

Finally, calculate cost from the manifest. Include stored GB-months, PUT, GET, LIST, and retrieval requests, data transfer, minimum retention, replication, and operational labor. A 2026 egress headline may be economically compelling, yet the service with cheaper transfer can still cost more if it imposes expensive request or retrieval charges. Record results with timestamps and configuration snapshots so another engineer can repeat the test rather than merely read a score.

Comparing S3, MinIO, R2, and Cross-Cloud Options

Amazon S3 remains the broadest managed baseline for S3-compatible applications, with a large set of storage classes, replication, and integration options. It also introduces service-specific behavior and potentially complex regional pricing. MinIO provides self-managed object storage distributed under the AGPLv3 and can be attractive where teams require direct control over infrastructure, but hardware, upgrades, security, and recovery become customer responsibilities. Its API compatibility helps application portability, although deployment topology and client connectivity strongly affect measured performance.

Cloudflare R2 emphasizes zero internet egress charges, making it relevant to workloads with substantial public delivery. It is not a universal performance replacement for S3, and benchmarks must establish the specific operations and regions involved. Cross-cloud object-storage services and data-plane platforms can provide portability, unified observability, or routing across providers, but they add another layer whose overhead should be measured separately. If a platform routes every request through a regional proxy, record that proxy's CPU, bandwidth, and added latency rather than attributing the entire result to the backing stores.

MLCommons storage benchmarks offer a more structured reference for storage-system testing, including attention to small objects and MLPerf Storage scenarios. MLPerf Storage v3.0 results, together with later extensions covering S3 and workloads such as vector databases and KV-cache use cases, show why storage benchmarks are becoming more workload-specific. A database logo on a result is not a complete storage benchmark, however. The public result must provide a reproducible method, participating-system configuration, and clearly defined workload before it is used to support a procurement decision.

No candidate should win solely because of an uploaded score. Weight p99 latency according to the user-visible target, sustained throughput according to the batch deadline, and cost according to the actual monthly request mix. Include contractual limits, support response targets, durability claims, recovery behavior, and exit feasibility. Portability is valuable only if the application can operate normally when the preferred provider is unavailable or a contract changes.

Common Benchmarking Mistakes That Distort Decisions

The most common mistake is changing several variables at once. Different client hardware, regions, encryption settings, test datasets, and concurrency make two numbers incomparable even if both are labeled “S3 API.” Another mistake is reporting only mean latency. With 99% of requests completing in 40 ms and 1% taking 4 seconds, the average can look acceptable while tail-sensitive users experience severe delay. Publish the sample count, percentile method, error count, and whether timed-out operations remain in the distribution.

Teams also frequently forget request charges. A workload moving 1 TiB may make storage capacity appear dominant, while a workload handling 100 million 4 KiB objects can generate substantial request expense and metadata pressure. Zero egress is helpful but does not eliminate every cost, and provider claims should be checked against current pricing. Comparisons can become stale quickly, which is why a September 2026 evaluation should include the retrieval date and region rather than reuse a 2023 price table.

Retry handling requires equal scrutiny. Disabling retries on one system while allowing aggressive retries on another rewards whichever hides failures best. Define a common client deadline, retry ceiling, and backoff policy, but also report raw first-attempt results. Verify objects after transfer with checksums or application validation, since a successful HTTP response does not by itself prove end-to-end correctness. Short tests with warm caches and pre-existing objects can measure storage cache behavior rather than the expected cold or cold-ish path, so describe the state accurately.

Finally, avoid treating a generic benchmark as a production simulation. AI training, backup, media processing, and database attachment workloads have different bottlenecks, and newer AI workloads can place unusual demands on metadata and small-file access. A credible report separates these cases instead of averaging them into a single number. It also acknowledges the limits of the dataset, the test period, and the regions examined.

Cost, Pricing, and When Platform Teams Should Act

Object storage pricing normally combines capacity, requests, and transfer. AWS S3 Standard has historically been listed at US$0.023 per GB-month in US East, with separate request and data-transfer charges; those figures are a baseline, not a universal quote. Cloudflare R2 has used zero egress pricing alongside per-million-class request fees, while MinIO's software license does not include the cost of servers, networking, support, monitoring, or administration. Total ownership therefore matters more than the license or headline storage rate alone.

Calculate cost per 1 million operations and per completed job, not merely cost per TB. For a 365-day retention policy, GB-month consumption is predictable, but lifecycle transitions, minimum durations, early deletion, and retrieval can alter the bill. For an egress-heavy application, zero-egress storage may justify migration testing even if request pricing is higher. For steady private ingestion, network architecture and committed capacity may matter more than public egress, so the strongest economic candidate can differ.

Act on benchmark evidence when a workload violates a stated service target, such as a p99 over 500 ms, sustained throughput below 70% of the application requirement, or an error rate above 0.1%. Those numbers are examples of decision thresholds, not universal standards; define thresholds from user experience and workload deadlines before seeing the results. A 1% error rate is generally unacceptable for many production pipelines, while a 0.1% error rate can still be intolerable for durable financial or scientific data.

Start with a two-provider proof of concept, then expand only if the first result survives application testing and cost review. Preserve the winning test corpus and scripts for later regression checks. For cross-cloud SaaS, evaluate whether portability and centralized observability save enough engineering time to offset routing overhead, and establish a fallback before migration. Benchmarking is most valuable when it supports a reversible decision with documented evidence, not when it serves as a permanent substitute for operational judgment.

The Evidence Standard for a Defensible Recommendation

A defensible recommendation contains enough detail for another team to reproduce it. It should name the providers, regions, object sizes, concurrency, duration, client hardware, network path, driver versions, and application workload. It should show p50, p95, and p99 latency, throughput, errors, retries, and a cost model. It should also disclose that one configuration does not represent every region or workload, especially for distributed object systems where node placement, replication, and network topology affect outcomes.

Treat third-party testing as either a reproducibility check or a source of new hypotheses, depending on its documentation. Studies of small-object performance across cloud storage services can be useful for screening, while ClickHouse-on-Minio-versus-S3 tests can reveal an application-level bottleneck. MLCommons material provides more formal benchmark context, and vendor documentation supplies current API and pricing facts. None automatically answers your internal question because your code, dataset, geography, and tolerance for operational complexity determine the right result.

The final recommendation should therefore be conditional. A managed service may be the practical choice for a team wanting fewer infrastructure duties, while self-managed MinIO may fit an organization with mature storage operations and a need for direct control. A cross-cloud data plane may help avoid permanent coupling, but only if its performance envelope and failure behavior are acceptable. Re-run the benchmark after meaningful changes to region, encryption, networking, driver, or retention, and review public comparisons when standards such as MLPerf Storage evolve.

The authoritative answer is not “which object store is fastest?” It is “which measured configuration meets our workload, reliability, and cost requirements?” Build the test around that question, keep the evidence visible, and reject any result that cannot be repeated.