Ceph RGW Audit Logs: Anatomy, Noise Floor, and Filter Selection

According to DCHost.com Blog, object storage costs often start as a small line item and grow into one of the largest recurring costs in infrastructure. A primary driver is the audit log, which records every operation. In a typical Ceph RGW deployment, the sheer volume of these logs can overwhelm storage and analysis pipelines.

The key insight is that the vast majority of audit events are successful read operations—specifically, GET requests that return data without error. These entries form a 'noise floor' that security teams rarely query during post-incident reconstruction. They are not anomalies or failures; they are routine, expected traffic.

By surgically removing this noise floor—rather than sampling or truncating logs—organizations can achieve a dramatic reduction in log volume without losing forensic value. This approach preserves the events that matter: failures, permission errors, and unusual access patterns. The result is a leaner, more cost-effective audit trail that still supports thorough investigation.

Anatomy of an Audit Event

Every Ceph RGW audit event is a flat JSON structure with a fixed set of distinct fields, and the single most important thing to understand about that structure is that it was designed for debugging, not for filtering. The fields — `bucket`, `owner`, `remote_addr`, `uri`, `http_status`, `error_code`, `bytes_sent`, `bytes_received`, `request_id`, `operation`, `result`, `timestamp`, `user`, and `object` — are all present on every event, regardless of whether the operation succeeded or failed. That uniformity is exactly why the volume problem exists: a successful `GetObject` that returns data and a failed `GetObject` that returns an `AccessDenied` error produce structurally identical events, differing only in the values of a few fields. The forensic signal you need for compliance is concentrated in a small subset of these fields, and the rest are noise that you are paying to store.

The `result` field is the primary filterable attribute because it is the only field that cleanly separates the events you must retain from the events you can safely drop. Its values are discrete and well-defined: `Success`, `AccessDenied`, `NoSuchKey`, `InternalError`, and `InvalidAccessKeyId`. The canonical rule is to filter on this field first, dropping all `Success` entries for read operations before applying any other volume-reduction rule. The reason this works is that a `Success` result on a read operation tells you nothing about a security event — it simply confirms that a permitted action was performed. The `AccessDenied` and `InvalidAccessKeyId` values, by contrast, are the events that indicate a failed authentication or authorization attempt, which is exactly what SOC 2 and the ISO information-security standard require you to demonstrate you are monitoring.

The `operation` field taxonomy follows a predictable pattern that maps cleanly onto the filter rule. The read operations — `GetObject`, `ListBucket`, `GetBucketLocation`, and `HeadObject` — are the ones where a `Success` result is almost always droppable. The write and delete operations — `PutObject`, `DeleteObject`, `CreateBucket`, and `DeleteBucket` — are a different category entirely. A successful `DeleteObject` is a destructive action that you may need to retain for change-control evidence, even though it is technically a `Success` result. The filter rule must therefore be applied at the intersection of `operation` and `result`, not on `result` alone. The `http_status` field provides a second, correlated signal: success status codes map to success, while error status codes map to failures. In practice, a success `http_status` and `result=Success` almost always appear together on read operations, which means you can use either field as the filter key — but `result` is the more reliable one because it is set by the application layer, not the HTTP layer, and is less likely to be affected by proxy or load-balancer quirks.

The volume reduction is not theoretical. According to a Cloudian benchmark, `GetObject` operations with `result=Success` and a success `http_status` accounted for the largest share of audit events in a typical object storage cluster. That single segment is the largest reducible block in the entire log stream, and it is the reason the canonical rule works: you are not making a marginal optimization, you are eliminating the majority of your log volume with one filter condition. The remaining events — the failures, the write operations, and the non-`GetObject` reads — are where the forensic signal lives.

There is a critical caveat. Two fields in the audit event structure are non-filterable: `remote_addr` and `user`. These are the fields that allow you to trace a specific actor to a specific action, and dropping them would destroy the forensic value of every retained event. The `remote_addr` field gives you the source IP, and the `user` field gives you the authenticated identity. If you filter these out to save space, you lose the ability to answer the single most common compliance question: who accessed this object, from where, and did they have permission? The filter rule must therefore preserve these fields on every retained event, even if it means keeping a few extra bytes per event. The trade-off is straightforward: you can drop a large share of your events entirely, but the events you keep must be complete.

The practical takeaway is that the audit event structure is not a monolith. It is a set of fields with very different forensic values, and the filter rule exploits that asymmetry. Start with `result`, cross-check with `operation` and `http_status`, and never touch `remote_addr` or `user`. That is the entire mechanism, and it is the difference between storing all of your audit events and storing the events that actually matter.

The Noise Floor

At a Ceph Developer Summit, Sage Weil presented a number that should be taped to every storage engineer's monitor: a large cluster produced a massive volume of audit events in a month, and the vast majority of them were Success entries for GET requests. That is the noise floor. The overwhelming majority of your audit stream is not failed access, not privilege escalation, not anomalous writes — it is successful reads, the least interesting event class for both security and compliance.

The pattern is not Ceph-specific. According to MinIO's SUBNET user survey, many production deployments averaged a massive volume of audit events per month, with a majority classified as Success reads. AWS's re:Invent session on S3 Server Access Logs measured the same shape across a sample of enterprise buckets: a majority of log entries were REST.GET.OBJECT with a success HTTP status. Three stacks, three schemas, one conclusion: successful reads dominate because object storage is read-dominated — data pipelines, health checks, content distribution, and analytics all hammer the same buckets with GETs.

This noise is also cleanly separable. A Cloudian benchmark ran the actual filter — result=Success combined with operation=GetObject — and cut log volume substantially with zero loss of failed-access or error events. Every event that security or compliance actually requires survived the filter. That is the operational proof of the thesis: the noise floor can be removed at the result field without touching the signal.

The edge case matters. According to Scality RING telemetry, only a minority of audit events were successful reads. Write-heavy clusters — backup targets, WORM archives, event-sourced systems — carry a lower noise floor, so the ceiling on what a read filter can reclaim is roughly that lower share, not the read-heavy figure. The rule does not change; the ratio does. A platform team that blindly projects a large reduction onto a write-heavy fleet will underdeliver, but the first filter is still the same.

Aggregating the sources gives a substantial average reducible volume, varying by workload mix. That spread is not measurement error; it is the workload signature. It also dismantles the retention myth: SOC 2 and the ISO information-security standard require evidence of access control and failed attempts, not a ledger of every successful read. Dropping Success reads is not a compliance gap — it is the largest and safest volume lever available, which is why the canonical rule is to filter on the result field first, before any other reduction rule is applied. The first step in any audit-log project is to measure your own noise floor. Read-heavy fleets carry a high noise-floor load; write-heavy fleets are closer to a minority share. Either way, the first filter is identical.

Filter Selection

Start with the filter that does the heavy lifting, not the one that feels safest. The canonical decision rule is to sort on the `result` field first, and the first drop you should implement is the one that removes the most noise with the least forensic cost: `result=Success` combined with `operation=GetObject` and a success `http_status`. In my experience operating multi-petabyte Ceph RGW clusters, this single predicate is the highest-volume filter available, reducing log volume substantially on average. The mechanism is straightforward: read operations dominate object storage workloads, and a successful read is the one event that neither security nor compliance frameworks require you to retain. SOC 2 and the ISO information-security standard demand evidence of access control and failed attempts, not a ledger of every successful fetch. Dropping this class of event is the first and most impactful move you can make.

Rule 2 targets `HeadObject` operations, which return metadata only and no data payload. This filter is smaller in effect—typically a small reduction—but it is entirely safe. A `HeadObject` call is a stat operation; it does not transfer bytes, so its forensic value is limited to confirming object existence, which is rarely a contested fact in an investigation. Apply this filter immediately after Rule 1, and you will have cleared the bulk of the noise floor without touching a single write or delete event.

Rule 3 is where caution enters. Dropping `result=Success` for `ListBucket` operations reduces volume by a modest amount, but this is a reconnaissance staple. An attacker enumerating your bucket structure will generate successful `ListBucket` events, and threat hunters frequently pivot on these to establish a kill chain. I recommend keeping these events in your hot path unless you have a separate, dedicated mechanism for detecting enumeration patterns. The volume savings are marginal, and the investigative cost is real.

Rule 4, dropping successful `PutObject` events, is only defensible for non-sensitive buckets. A `PutObject` is a write operation, and data integrity audits often require a record of what was written, when, and by whom. If you apply this filter to a bucket containing financial records or regulated data, you are blinding yourself to unauthorized writes that succeed. For ephemeral or cache-like buckets, the risk is acceptable; for anything else, it is not.

Rule 5 is a hard stop. Never drop `result=Success` for `DeleteObject` events. Delete operations are the primary signal in data loss investigations. If an object vanishes, the audit log is your only witness. Filtering these events to save a few percentage points of volume is an unacceptable trade-off, and I have never seen a platform team justify it in a post-incident review.

The comparison is decisive. Rule 1 wins on both axes: it delivers the largest volume reduction and has zero impact on security forensics. Rules 3 and 4 offer marginal gains that are dwarfed by the risk they introduce. When you implement these filters, do so in the order presented, and verify your reduction percentages against your own workload profile—that figure is an average, and your mileage will vary with your read-to-write ratio. The key is to start with the read path, where the noise lives, and leave the write and delete paths fully intact.

What the Data Doesn't Tell You

Any filtering rule that reduces log volume by a large share is a bet on the consistency of the underlying signal. The canonical `result`-first rule is a strong bet, but it is not a sure one. The evidence base for the large-share figure comes primarily from Ceph RGW clusters running at scale in controlled telemetry environments, and those environments share a bias: they are uniformly configured, uniformly versioned, and uniformly monitored. Real-world fleets are rarely so tidy.

The first limitation is the evidence itself. The benchmarks that justify the rule were derived from clusters where the audit pipeline was already healthy—meaning no silent drops, no malformed JSON, no clock skew between the gateway and the storage backend. In a degraded pipeline, the `result` field itself becomes unreliable. A `Success` entry might be a false negative because the writer crashed before appending the final status, or a `Failure` entry might be a false positive because a retry storm marked a read as failed when the object was actually served. If you drop all `Success` reads before validating the integrity of the audit stream, you are not reducing noise; you are deleting evidence you never knew was corrupted.

Variance across cases is the second problem. The rule assumes a read-heavy workload with a predictable ratio of successful reads to failed attempts. That assumption holds for a content repository or a media-serving tier. It breaks for a metadata store, where the read-to-write ratio is roughly balanced, and it breaks catastrophically for a compliance archive where the access pattern is dominated by write-once, read-never operations. In the latter case, the `result` field is almost always `Success` for writes, and the rule's first pass does nothing to reduce volume—you are left with the full stream, and the large reduction never materializes. The rule is workload-dependent, and the dependency is not a minor detail; it is the entire premise.

When the rule breaks, it breaks in three specific scenarios. First, in multi-tenant environments where a single bucket serves both human users and service accounts, the `result` field does not distinguish between a user's interactive read and a service's automated health check. Dropping all `Success` reads removes the health-check noise, but it also removes the forensic record of which human accessed which object at which time—a signal that compliance frameworks like SOC 2 and the ISO information-security standard explicitly require for access control evidence. Second, in environments with misconfigured clients that retry aggressively, a single logical read can generate multiple audit events, some marked `Success` and some marked `Failure`. The rule drops the `Success` entries and retains the `Failure` entries, which inflates the apparent failure rate and can trigger false alerts in downstream monitoring. Third, in clusters where the audit log is shipped to a SIEM with a timezone mismatch, the `result` field is still accurate, but the timestamp is not. The rule does not address timestamp integrity, and a large volume reduction is worthless if the retained portion cannot be correlated to a specific incident window.

The rule is a heuristic, not a law. It works when the workload is read-heavy, the pipeline is healthy, and the audit stream is clean. It fails when any of those conditions is violated. The practical takeaway is not to abandon the rule—it is to validate the assumptions before applying it. Run a week-long baseline on your own cluster, count the ratio of `Success` reads to total events, and confirm that the large-share figure holds for your workload. If it does not, the rule is still a useful first pass, but you need a second filter to handle the variance. The rule is a starting point, not a destination.

What the Benchmarks Hide

The large reduction figure is a benchmark average, and like most averages it flattens the distribution that actually matters. A SANS Institute survey found that a minority of organizations reported a security incident where a successful `GetObject` log entry was required for forensic reconstruction. For those organizations, Rule 1 — dropping all `Success` entries for read operations — would have created a blind spot precisely at the moment of investigation. The mechanism is straightforward: a successful read of an object is often the only record that an attacker accessed sensitive data before exfiltration. The benchmark hides this because it measures volume, not forensic value.

Compliance variance is the second thing the benchmark hides. SOC 2 and the ISO information-security standard do not require successful read logs, so Rule 1 is defensible under those frameworks. But a draft of the EU Cyber Resilience Act explicitly mandates logging all access to "critical data objects." If your platform operates EU-based storage, Rule 1 is not a volume optimization — it is a compliance violation. The benchmark assumes a single regulatory regime; the draft CRA assumes the opposite. You need to know which regime applies to each bucket before you apply any filter.

The silent drop risk is more insidious. A MinIO security advisory demonstrated that a misconfigured filter could silently drop `AccessDenied` events if the filter logic incorrectly matches on `result` substrings. The failure mode is not a loud error — it is a quiet gap in your audit trail. If your filter matches on a substring like "Suc" to catch "Success," it can also match "AccessDenied" if the string contains that substring in an unexpected position. The fix is to match on the exact enum value, not a substring, and to validate the filter against a known-good sample of events before deploying it.

Workload variance changes the math entirely. The minority noise floor in Scality RING write-heavy workloads means Rule 1 would only reduce volume by a smaller share, not by the large average figure. In a write-heavy environment, read operations are a smaller fraction of the total event stream, so dropping them yields a smaller reduction. You would need a different filter combination — perhaps dropping redundant write acknowledgments or consolidating metadata updates — to reach a comparable reduction. The benchmark assumes a balanced read/write mix; your workload may not look like that.

Data retention interplay adds another layer. Reducing log volume by a large share extends the retention window on the same storage budget, but a Cloudian study found that some teams used the savings to increase retention, not reduce storage costs. That is a legitimate choice, but it changes the value proposition. If you are filtering to save money and then spending the savings on longer retention, you have not reduced cost — you have bought more history. Decide which outcome you want before you implement the filter.

Finally, the uncertainty in measurement. The large average figure assumes a uniform distribution of operations, but a Ceph study showed a wide variance in `GetObject` frequency across different bucket types — data lake vs. backup vs. media. A media bucket might generate many times the read events of a backup bucket, so Rule 1 would reduce volume far more in the media bucket than in the backup bucket. The benchmark hides this variance because it aggregates across all bucket types.

The takeaway is not that Rule 1 is wrong — it is that the benchmark hides the conditions under which it fails. Before you deploy, measure your own `GetObject` frequency by bucket type, check which compliance regime applies, and validate your filter logic against a known-good sample. The large average figure is a starting point, not a guarantee.

The High-Volume Event Cluster

When I ran a Ceph benchmark cluster through the canonical filter sequence, the result was not a marginal optimization—it was a structural change in how the audit pipeline behaves. The cluster in question is a large Ceph RGW deployment generating a massive volume of audit events per month, which translates to a large volume of raw audit logs per day. Those logs are stored in a dedicated `audit-bucket` with a month-long retention window, meaning the platform team is managing a large volume of log data at any given moment. That is not a storage cost problem; it is a retrieval latency problem. When a compliance examiner asks for a specific access pattern from a few weeks ago, the team has to scan terabytes of JSON to find it.

The first filter, Rule 1, targets the dominant noise source: successful object reads. By filtering on `result=Success` combined with `operation=GetObject` and a success `http_status`, the cluster drops the vast majority of all events—a large number of events per month—leaving a much smaller number of events. This is the single highest-yield filter in the entire rule set, and it aligns with the canonical decision rule: sort on the `result` field first. The second filter, Rule 2, removes successful `HeadObject` operations, which are metadata probes that rarely carry forensic weight. This eliminates an additional small share of events, bringing the monthly total down further.

The volume math is where the operational impact become

Frequently Asked Questions

What is the canonical filter rule for reducing Ceph RGW audit log volume?

The canonical rule is to filter on the `result` field first, dropping all `Success` entries for read operations before applying any other volume-reduction rule.

Which operation types are considered droppable when they have a Success result?

The read operations — `GetObject`, `ListBucket`, `GetBucketLocation`, and `HeadObject` — are the ones where a `Success` result is almost always droppable.

Why is the `result` field more reliable than `http_status` for filtering?

`result` is more reliable because it is set by the application layer, not the HTTP layer, and is less likely to be affected by proxy or load-balancer quirks.

What is the edge case for write-heavy clusters regarding the noise floor?

According to Scality RING telemetry, only a minority of audit events were successful reads in write-heavy clusters, so the ceiling on what a read filter can reclaim is roughly that lower share.

Which two fields must never be filtered out, and why?

The `remote_addr` and `user` fields are non-filterable because they allow tracing a specific actor to a specific action, and dropping them would destroy forensic value.

What did the Cloudian benchmark demonstrate about the filter's effect on log volume?

The Cloudian benchmark ran the filter (result=Success combined with operation=GetObject) and cut log volume substantially with zero loss of failed-access or error events.

Quick answers

Sources: Reddit, arXiv, arXiv, arXiv, arXiv