The Future of Logs: Rectifying Names

If names are not correct, language will not be in accordance with the truth of things; if language is not in accordance with the truth of things, affairs cannot be carried on to success. — Analects, Book XIII (Zilu)

Preface: Lately I’ve been building a serverless framework. The deeper I go, the more convinced I become that logging will be the life-or-death crux of the whole thing. Every single serverless invocation casually spits out a dozen or so log lines; if I follow the traditional practice and record them all faithfully, the moment traffic picks up, observability alone will blow the AWS bill into astronomical figures. I sprawl on scratch paper sketching fields, calculating retention, estimating cardinality, and keep asking myself the same question: does logging really have to be strings?

The End of the String Foundation

Open any service written with tracing / log4j / zap, and you’re looking at the same fundamental thing:

info!(user_id = %uid, action = "retry", "after upstream failure");

It looks like data, but in its bones it is a string. The key user_id doesn’t exist anywhere—it exists only in this one call site. The next engineer who copy-pastes it might write user, userId, uid, or u_id, and nothing will stop them at the compiler. When launch day finally comes, every PromQL, Lucene, and SQL query you run in Loki / Datadog / Splunk rests on these word-of-mouth field names.

This is the true state of logging today: no schema, only convention; no constraints, only tacit agreement. A “tacit agreement” hammered out by hundreds of engineers, a dozen languages, and thousands of call sites is destined to be worn away by time. Refactors, renames, merges, migrations—each one quietly tugs at some thread in this agreement. That monitoring still works is pure survivorship bias.

What’s even more painful is cardinality. One careless request_id = %rid written into a metric label plants a bomb right then and there in Prometheus, and it may not detonate in the in-memory map until some day next month. Every team that has used Prometheus has lived through this at least once; afterward everyone swears never to do it again, and six months later a newcomer does it again. The reason isn’t that engineers are dumb—it’s that the cardinality of a label is never expressed in the code. The compiler can’t see it, CI can’t see it, PR review can’t see it either. Once cardinality climbs, the only thing that can tell you is a 3 a.m. memory alert.

The third headache is cost. On a multi-tenant platform, log volume is terrifying. A while back I did a rough calculation on my own framework: an average function invocation casually produces a dozen log lines, all shipped to object storage. Just storing this pile of “info / debug text” costs five figures a month in dollars—and less than 1% of it ever gets queried by a human. The other 99% is set aside for “just in case,” and “just in case” never comes.

String logs have three chronic diseases: silent failures during evolution, unguarded cardinality, and costs that invert value. This is not a problem with any particular product; it’s a problem with the data shape itself. Ten years ago Splunk / ELK / Loki chose the path of “free text + indexing” because at the time that was the only path that got people productive quickly; we keep walking it today out of inertia.

It’s time to ask: in the age of AI, if we redesigned from scratch, what should logs look like?

Give Schema Back to Observability

My answer is blunt: define all logs, metrics, and audit events in protobuf; code may only emit messages that have already been defined. Those loose, unbundled logs in the ecosystem that aren’t ours get taken in by a single catch-all message.

To explain, let me use the simplest possible todo app as an example. In proto/todo/v1/events/, place an event catalog:

syntax = "proto3";

package todo.v1.events;

import "todo/obs/options.proto";

// Event: a todo was created
message TodoCreated {
  option (todo.obs.tier) = TRACE;
  option (todo.obs.retention_days) = 30;

  string list_id  = 1 [(todo.obs.label) = true];
  string priority = 2 [(todo.obs.label) = true];   // "low" | "med" | "high"
  string source   = 3 [(todo.obs.label) = true];   // "web" | "ios" | "api"
  string todo_id  = 4 [(todo.obs.tier)  = TRACE];   // high cardinality, goes to trace
}

// Event: sync to cloud failed
message TodoSyncFailed {
  option (todo.obs.tier) = LOG;

  string list_id         = 1 [(todo.obs.label) = true];
  ErrorCategory category = 2 [(todo.obs.label) = true];
  string error_message   = 3 [(todo.obs.cardinality) = UNBOUNDED];
}

At the call site, only a macro remains:

emit!(TodoCreated {
    list_id:  list_id,
    priority: "high",
    source:   "web",
    todo_id:  todo_id,
});

Misspelled a field? Compile error. Left out a required field? Compile error. Trying to use todo_id as a label? Compile error, because its tier is TRACE, and the macro routes it to trace attributes rather than metric labels. The entire “form” of observability is raised from a line of string to a schema that carries version, owner, and comments.

This sounds like putting shackles on developers. But these shackles are precisely the bone that observability has been missing for a decade. Let me break down the benefits one by one.

Old Pain Points Vanish at Compile Time

Evolution Semantics Come Free from Protobuf Itself

Protobuf’s schema evolution rules are astonishingly mature: adding fields is backward compatible, deleting fields requires reserving the tag first, renaming is an explicit two-step migration, and versioning the whole message just means renaming it to TodoCreatedV2. These rules aren’t new—they’re the engineering wealth that Google accumulated over twenty years of stepping on land mines.

Apply it to observability and several old pain points vanish outright:

• Renaming a field? That breaks the schema; CI refuses to merge.
• Adding a field? Additive; old consumers don’t even notice.
• Deleting a field? It must first be marked deprecated; every dashboard / alert reference gets scanned out by tooling, and only then does removal go through.

Most critically, dashboards, alerts, and alert rules now reference the schema directly instead of strings. Rename one field and the dashboard CI goes red immediately. This feedback loop used to close only when the on-call engineer got woken up; now it closes at PR review time.

Cardinality Goes from “Runtime Bomb” to “Compile-Time Constraint”

Protobuf’s option mechanism gives cardinality, for the first time, a place to be declared:

string list_id   = 1 [(todo.obs.cardinality) = LOW];        // label
string priority  = 2 [(todo.obs.cardinality) = LOW];        // label
string todo_id   = 3 [(todo.obs.cardinality) = UNBOUNDED];  // attribute
string user_id   = 4 [(todo.obs.cardinality) = HIGH];       // trace attribute

Given the schema, a lint tool in CI can do several things that runtime can never do:

• Compute the Cartesian product of all LOW fields to estimate the number of label combinations, and reject the merge if it exceeds a budget (say, 100k).
• Route UNBOUNDED fields to the trace or log layer, and never let them appear in a metric label.
• Keep a cardinality ledger per tenant and intercept event schemas that exceed quota before they ship.

This is a paradigm shift. In the past, cardinality issues were solved by a human squinting at the “top series count” on a dashboard, or by ops cleaning up after the fact. Now the problem is solved at compile time.

Pre-Aggregation Goes from “Runtime Optimization” to “Structural Property”

If every emit is a typed proto message, a node can absolutely pre-aggregate in memory locally:

counters[(TodoCreated, list_id=X, priority=high, source=web)] += 1

String logs could also do this, but only via sampling, via regex, via ops guessing which logs are aggregatable. Guess wrong and you lose real data; guess right and it’s still incomplete. Pre-aggregation over proto is far more straightforward: low-cardinality fields form the key, the counter increments by one, a delta is reported once per scrape interval, and the event volume on the wire drops by two to three orders of magnitude outright.

My rough estimate: for a mid-scale system producing 5TB of raw logs a day, reworking it along the “aggregate low-cardinality fields + send high-cardinality fields separately as trace” scheme would compress landed storage down to somewhere above 50GB—90% of it trace, 10% of it metric counter deltas. An order of magnitude in cost savings, all because the schema tells the system up front “what’s skeleton, what’s flesh.”

Audit, Compliance, and Cross-Language

Audit

Audit events are the strongest fulcrum for schema-first. Audit inherently demands strict fields, strict retention policies, and strict classification. How do most companies do it today? Ops writes a wiki page of “audit field conventions,” every team implements it themselves, and six months later things start to drift; then a compliance audit comes along and triggers another spring cleaning.

Write it in proto and audit degenerates into a chunk of schema:

syntax = "proto3";

package todo.v1.audit;

message TodoAuditAction {
  option (todo.audit.retention_days) = 2555;        // 7 years
  option (todo.audit.classification) = PII;

  string actor   = 1 [(todo.audit.field) = SUBJECT];
  string list_id = 2 [(todo.audit.field) = RESOURCE];
  string action  = 3 [(todo.audit.field) = ACTION];   // CREATE | UPDATE | DELETE | SHARE
  AuthzDecision decision = 4 [(todo.audit.field) = OUTCOME];
  string source_ip = 5 [(todo.audit.cardinality) = UNBOUNDED];
}

Compliance auditing stops being “dig through 500 microservices’ log call sites” and becomes “review the proto directory.” This is what cloud providers, big tech, and virtually every financial system already do internally; most SMBs simply lack the capability or energy to walk that path.

Cross-Language Becomes a Byproduct of Codegen

In the framework I’m building, the host is Rust, and user functions must support JS, Python, and Go. Every additional language SDK used to force reinvention of a logging convention, a metric helper, and an audit client. After proto-first, each additional language just means: run protoc once, wrap a thin syntactic layer, and every event, every dashboard, every alert is immediately universal.

One schema, many bindings—this is the dividend protobuf already proved in RPC. Extending the same mechanism to observability is going with the current, not building a new dam.

Self-Documenting Event Catalog

protoc --descriptor_set_out builds a descriptor for the entire event catalog. This descriptor can be fed to:

• Grafana: auto-generate dashboard templates.
• Alertmanager: auto-generate alert rule skeletons.
• Docs site: proto comments become event documentation directly.
• IDE: full field completion when emitting.
• Compliance systems: one-click export of “which audit events does this service emit.”

This pattern of “one descriptor, many tools consuming it”—OpenAPI achieved it for APIs, proto achieved it for RPC, yet observability alone has been limping along on strings. Give schema back to observability and you give back all that automation potential along with it.

Why Wasn’t This Done Before

Now that I’ve finished listing the benefits, let’s talk about the drawbacks.

Regression in Developer Ergonomics

Today, typing info!("retry happened for user={uid}") takes five seconds. With schema-first: open the proto file → add a message → run codegen → write the emit call. If you also need to review cardinality, PR review goes another round. That’s a tenfold burden.

Engineers vote with their feet. The most common escape route is to define an all-purpose DebugEvent { string payload = 1; } and shove every field they don’t want to schema-ize into it. Schema-first gets beaten back into its original form within a week.

As I’ll discuss later, this pain point nearly vanishes in the agent era. But even today, it’s not a dead end: you need a controlled dev channel (dev_event!) that fails to compile in production; usable during debugging, but must be seriously reconsidered before release. Trading short-term friction for long-term governability is a fair deal.

Schema Bloat

A mid-scale codebase will have thousands of distinct “logging intents.” If every one is promoted to a proto message, over a few years the event catalog could balloon to tens of thousands of messages. Proto compilation slows down, repo file count balloons, duplicate definitions become inevitable.

There’s no way around this pain point, only two mitigations:

• Modularize into files: todo/v1/events/list.proto, todo/v1/events/sync.proto, todo/v1/events/auth.proto—split by module to localize the blast radius.
• Abstract common patterns: HttpCallFailed, RetryAttempted, ResourceExhausted—turn these scenarios into reusable event skeletons, so you don’t invent a new message for every retry site.

“Slimming down” the event catalog is continuous work. But compared to the “dark growth” of string logs (you can’t see it, but it grows with every deploy), proto’s growth is explicit, visible, and periodically reviewable.

The “Unknown Unknowns” at 3 A.M.

This is the one that makes me most hesitant.

Roused at 3 a.m., you don’t know what to look for. The sheer vastness of logs can bail you out—e.g., you can grep -i "slow query" node.log | grep alice and try your luck. Any field, any wording, any accidentally printed stack trace could become the clue that cracks the case. This “unknown unknowns” wealth is what string logs accidentally accumulated.

Proto-first severs that path. You can only query events that were already defined. If some new error path added last week never got an event defined, today’s outage has zero signal. Schema-first has 100% coverage of the known failure domain and 0% coverage of the unknown failure domain.

This is the answer schema-first must provide, and it leads to what I want to discuss in the next section: keep an escape hatch.

Third-Party Libraries

Your code can be all emit!, but upstream crates like reqwest, tokio, sqlx will not—they all uniformly tracing::warn!. Schema-first’s purity breaks the moment you add your first third-party dependency.

There’s no perfect answer, but there is a workable one: install a tracing subscriber that captures all non-proto tracing events, wraps them into a unified LegacyTracingEvent, and treats them as the “log tier” (short retention, strict rate limiting, excluded from metrics). Third-party observability is forever a second-class citizen; you must accept that.

ForensicEvent: The Escape Hatch

For schema-first to be truly usable, several controlled bypasses must exist alongside the main trunk. I call them “escape hatches.” The most important one is ForensicEvent.

syntax = "proto3";

package todo.v1.events;

message ForensicEvent {
  option (todo.obs.tier) = LOG;
  option (todo.obs.rate_limit_per_min) = 1000;
  option (todo.obs.retention_days) = 7;

  string component = 1 [(todo.obs.label) = true];
  string category  = 2 [(todo.obs.label) = true];
  string trace_id  = 3 [(todo.obs.tier)  = TRACE];
  string payload   = 4 [(todo.obs.cardinality) = UNBOUNDED];
}

Its design philosophy comes down to a few sentences:

1. It is part of the schema, not the antonym of schema. It has a name, an owner, a retention policy, and a rate limit. It appears in the event catalog, can be audited, monitored, and its usage counted.
2. It emits only on error paths by default. Healthy paths go through defined events; on error paths, after emitting the normal event, you can attach a ForensicEvent to capture the on-site state.
3. It is coupled to trace sampling. For a failing request, every ForensicEvent emitted along its path is retained; for a healthy request, every ForensicEvent on its path is dropped. This preserves the crime scene without blowing up storage.
4. Its usage count is a first-class metric. Which module emits how many ForensicEvents per day, what category dominates—all visible. If some module starts treating it as a universal log bucket, the dashboard sees it first, giving governance a handle.

With ForensicEvent in place, adding two more bypasses completes the picture:

• The dev_event! macro: available in dev / debug builds, a hard compile error in release builds. Use it while exploring code and fixing bugs; must be upgraded to a predefined event before release.
• LegacyTracingBridge: uniformly absorb third-party library tracing events into a single message and drop them at the log tier.

These three bypasses together don’t circumvent schema-first—they manage everything outside the schema the schema way. It names the “unknown” rather than letting the “unknown” run wild.

With ForensicEvent in place, the shape of “can’t find the logs at 3 a.m.” changes: you’re not left with nothing—you get a ForensicEvent stream with on-site snapshots from the error path. And once the outage is diagnosed, the very next PR adds the corresponding event to the schema, so this class of failure produces structured signal the next time. The event catalog grows organically along the direction of “failures that have occurred,” and the schema evolves in step with production.

The Reversal in the Agent Era

If you’ve read this far and still think the approach is “idealistic but too heavy,” let me share something that changed my mind: almost all the “pain points” of this approach are pain points for human engineers; once the primary code author goes from humans to agents, most of the pain disappears and the value multiplies.

Let’s revisit those pain points from the last section:

Development burden. Opening a proto file, adding a message, running codegen, writing emit—for a human, that’s a thirty-second context switch; for an agent, it’s one tool call. Agents have no “flow state,” no “can’t be bothered to get up,” no “eh, let’s just println for now.” They define every event cleanly, because defining things is their most natural working mode.

Schema bloat. Thousands of messages are a burden to a human but data to an agent. “Is there already an event in the catalog that duplicates my current intent?” Humans answer with their brain and grep; agents answer by semantic matching—the latter is an order of magnitude more accurate. Bloat still has to be managed, but dedup, merging, and categorization—an agent does all of these more reliably than any dedicated “observability PM.”

Debug-time friction. Humans like ad-hoc println because println is cheaper than adding a schema. For an agent, both are equally cheap. The dev_event! escape hatch is almost never needed when an agent writes code, because the main path is its fast path.

Multi-language. Adding one more guest SDK is weeks of engineering for a human, one template generation for an agent.

Governance. Reviewing all emit call sites, checking cardinality budgets, finding duplicate events, chasing deprecated fields—these “repetitive but detail-oriented” tasks are exactly an agent’s comfort zone.

When you get down to it, humans resist schema-first mainly because it means typing more keys and going through one more process. Agents don’t care about either—they just do it. So the question flips: today we tolerate string logs because they’re cheap for humans and expensive for the system; tomorrow, when agents write the code, schema is just as cheap for them, so why should the system keep bearing the expensive cost?

This is my judgment for the AI era: proto-first observability is shifting from “should but too expensive” to “obviously the right thing to do.” The inflection point isn’t some new tool release—it’s that the subject of code authorship silently changed.

A New Problem: Semantic Drift

In the agent era, after adopting schema-first, there is no shortage of new problems—they just change shape. The biggest one is semantics (understanding).

When an agent writes an authorization-failure branch, it emits AuthzDenied, when in reality it’s a case of the user not existing at all—it should have emitted UserNotFound. The schema is correct, the fields are all populated, but the event name misaligns with semantics. Every alert keyed on “AuthzDenied” starts firing false positives.

Two agents working in parallel PRs each define an event—one calls it RequestThrottled, the other RateLimited—describing the same thing, landing in different dashboards. Monitoring splits, and nobody notices.

When an agent refactors a piece of code, it judges some event to “look redundant” and deletes it. Every alert depending on that event goes quietly dark.

In the past, these problems were caught by “the code owner’s intuition.” A veteran sees a PR deleting AuthzDenied and frowns: “Hang on—dashboard X is still using this, isn’t it?” But an agent, absent the relevant context, has no such intuition. Its judgment is based on the code and lacks a semantic map of “what this event promised externally.”

This is the capability that the schema-first + agent paradigm truly needs to fill in. A few solutions:

• Events need their own tests. Since emit is a typed event, you can write a “behavioral contract” test for each event: when some precondition holds, the code under test must emit this event with fields satisfying some assertion. Agents are extremely good at writing tests; let them do this too.
• CI requires emit coverage. Every event in the catalog must be triggered by at least one test path. “Event is defined but never emitted in any test” goes red instantly. Useless events should not wander in the catalog.
• Semantic review is the last human gate. Syntax / fields / types / cardinality are enforced by machines, and agents check each other. But “is this event appropriately named? What does it promise externally?”—that judgment still deserves a human pausing to think for ten seconds. This isn’t micromanagement; it’s a semantic sanity check on the agent workflow.
• Deleting an event requires a second confirmation. Any agent-submitted PR that touches an event deletion automatically scans every dashboard / alert / downstream consumer and attaches the findings to the PR description. Make everything visible, then decide whether to click merge.

Compared to the human era, the problem shape changes from “nobody writes logs / writes the wrong logs” to “events misnamed / events semantically overlapping / events silently deleted.” These are harder problems to solve, but also more worthwhile ones. They point to “what do we want to get out of observability” rather than “how do we cobble observability together from strings.”

Start with Ten Events

Back to the serverless framework in my own hands—the rollout pace I have in mind is roughly this.

Step one: pick the ten highest-frequency internal events and proto them first. For example: RequestServed / RequestRejected / UpstreamFailed for user requests, FunctionInvoked / FunctionExited / ColdStartHappened for the runtime, and DeploymentApplied / BreakerTripped for the control plane. Declare their fields, cardinality, tier, and retention fully, migrate existing call sites, and run in production for a week or two. The purpose of this step is to draw the cost curve and give the team a “visible cost savings.”

Step two: build the emit! macro + three-tier routing (metric / trace / log) + cardinality lint. This is infrastructure; once done, all subsequent events benefit.

Step three: migrate audit. Audit already demands structure, so this step meets almost no resistance—and lets you close a batch of existing compliance gaps along the way.

Step four: define all metric emissions. Every place currently using the prometheus crate to directly increment counters now goes through emit!. This step is mechanical; an agent can handle it.

Step five: add ForensicEvent + dev_event! + LegacyTracingBridge. Once the escape hatch system is in place, the main path can be locked down tight.

Step six: wire obs-lint into CI. From here on, every change to the event catalog enters the governance process.

Step seven: slowly migrate the remaining long tail. No rush here—after the first six steps, 80% of storage and alerting chaos has already converged.

After these seven steps, the framework’s observability has a real “catalog”: dashboards auto-generated, alert templates auto-generated, tenant cardinality ledger online, and multi-language SDKs sharing one schema. From then on, every new feature lands in main by default with event schemas, and event definition becomes muscle memory.

The Next Stop for Observability

Treating logs as strings was a compromise of the previous generation of infrastructure. We’ve lived inside that compromise for decades, and we’ve already moved the foundation of the house onto shifting sand. If we don’t catch up on this schema lesson today, every observability bill, every alert incident, and every field some agent renames in the future will keep reminding us of the extra step we didn’t take back then.

Protobuf isn’t the only answer, and schema-first doesn’t necessarily require protobuf. But proto has run in the adjacent land of RPC for twenty years: engineering-stable, toolchain-complete, naturally cross-language, clean annotation mechanism. Extending it to observability is following the current, not founding a new religion.

The ForensicEvent escape hatch is the key to making this approach land. It tells the team two things: event definitions are the default; but we also have limited escape hatches. Anything without an event definition rides along in ForensicEvent; anything riding along too much, the dashboard will see for you; anything worth defining, the next PR will define. Schema grows, catalog evolves, and lives on alongside the product.

The agent era will push this onto the right track. The daily typing pain of human engineers was once the biggest resistance on this path; now that resistance is lifted. The remaining real problems—semantic drift, unknown unknowns, catalog governance—are a level higher; they are product problems, design problems, problems worth spending a human brain on.

Confucius’s answer to Zilu—“then it is necessary to rectify names” (必也正名乎)—reads today with an engineering precision of its own. The next stop for logs is to give every observed thing a name. Once the name is settled, language can be right; once language is right, affairs can succeed. That bill on scratch paper that made my heart pound has, for the first time, an answer I can actually land.