Log storage gets treated as a detail. You set a retention period, size the volume based on a rough estimate of log volume, configure the agent to write there, and move on. It’s not wrong as a starting approach — but it’s incomplete.
The incompleteness shows up at 10am on a Monday when the volume is full and the logging agent is doing something unexpected with write failures. In the best case, logs stop arriving and you notice. In a worse case, the agent’s write failure cascades into something that affects the node.
The thing you discover in that moment: log storage was always an architecture decision. You just hadn’t made it explicitly.
The decisions that matter
There are three questions about logging that have architectural implications, and most deployments don’t answer them explicitly.
Where do logs land, and what happens when that destination is unavailable?
Node-local log storage is simple and predictable. The agent writes to disk, a log shipper forwards to a central location, and the local copy ages out on retention. When the central destination is unavailable, the local copy buffers. When the local disk fills, the agent either drops logs or blocks — and that choice is a configuration decision that most deployments leave at the default without examining what the default actually is.
Centralised log ingestion — a managed service, an Elasticsearch cluster, a log aggregation platform — moves the storage question out of the node but creates a different dependency. If the ingestion endpoint is slow or unavailable, what backs up? Where? Is there a circuit breaker, or does backpressure propagate to the agent?
Neither architecture is wrong. Both need their failure modes designed, not discovered.
What’s the relationship between log volume and retention?
This one seems obvious and isn’t. Log volume under normal conditions and log volume during an incident are different. A system under stress generates more logs — error events, retries, health check failures, audit events from automated remediation. The spike in log volume happens exactly when you most need the logs to keep arriving.
A retention policy set against average volume will eat into the buffer faster during an incident than you expect. A disk sized to handle normal traffic will fill faster during the event you’re trying to debug.
The practical consequence: retention and disk size need to be calculated against peak volume, not average volume — and “peak” includes the volume generated during the types of events you’re monitoring for.
What’s the cost of losing logs in this environment?
In a regulated environment — financial services, healthcare, any platform with compliance logging requirements — losing logs during an incident is not just an observability problem. It may be an audit problem. The retention policy, the storage sizing, and the failure mode for write errors all need to satisfy a compliance constraint, not just an operational one.
On a securities brokerage platform in Indonesia, we used a Wazuh-based security event stack alongside Prometheus and Grafana for metrics. Security events and audit logs had different retention requirements from application performance logs — the former had defined minimum retention periods that were non-negotiable. Treating them identically would have been both wasteful and non-compliant.
The architecture decision: separate storage tiers with separate retention policies, sized independently against their respective volume and retention requirements.
The failure mode worth designing around
The scenario that sharpens thinking: a Fluentd or Fluent Bit agent with a disk write error and a configuration that has it retry indefinitely. The agent uses increasing CPU, the node becomes unresponsive to kubelet health checks, and the node is eventually evicted from the cluster. You’re now debugging a node availability problem that is actually a logging configuration problem.
This isn’t a theoretical failure mode. Logging agents have high privileges on the nodes they run on, and their default failure behaviour is often “keep trying.” Under disk pressure, that’s a combination worth thinking about before it happens.
The design decision: configure logging agents to drop rather than block under backpressure, with a metric that surfaces the drop rate. Dropping logs during a disk full event is a recoverable state. A node that goes unresponsive because the logging agent is blocking is not.
Drop is not ideal. Neither is losing the node. Choose which failure you’d rather have, and configure for it explicitly.
What this actually changes
Treating log storage as an architecture decision doesn’t mean overengineering it. It means:
Answering the backpressure question before deployment, not during an incident.
Sizing storage against peak volume, with a documented assumption about what peak means for this platform.
Separating log types that have different retention requirements rather than treating all logs as equivalent.
Configuring the failure mode for write errors explicitly — and making that configuration visible as a metric so you know when it’s activating.
None of those are large decisions. All of them are decisions that are harder to make retroactively when the disk is full and something is down.