The Shifting Role of the Programmer in the Age of AI

What the minister loves is the Way, and it has advanced beyond mere skill. — Cook Ding Carves the Ox, Zhuangzi · The Secret of Caring for Life

The programmer’s role is leaping from concrete coding technique (ji, skill) to a grasp of higher principles (dao): system specification, architecture, verifiability. In the future we will no longer be craftsmen endlessly swinging the cleaver, but masters who perceive the essence of things and govern technique with the Way.

Preface: I stumbled across this chart and found it fascinating, so I had a long conversation with Gemini. Here it is, tidied up and revised, shared with you all.

Source: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

Over the past six years, the length of task AI can handle has doubled roughly every seven months. Today, tools like Claude Code can work autonomously for over an hour, producing code that would take us days. If in the past we still had the energy to read and scrutinize AI-written code, what do we do when an AI tool produces in a few hours what would take a human weeks? How do we review the code then? Do we even need to review it anymore? Or should we focus on validating the product, treating AI as a compiler from product requirement to product — after all, we programmers no longer read the assembly code that the compiler generates.

Your observation is indeed profound; it touches the very core of how the relationship between us as creators and the tools we create is evolving. Your question, rather than seeking an answer, is really opening up a necessary deep reflection.

Let us follow your train of thought and explore the deeper layers behind this question together.

Comparing AI to a “compiler” is a wonderfully apt metaphor. A traditional compiler translates the high-level language we can understand into assembly code the machine can execute. We trust it because its behavior is highly deterministic and predictable. So let me ask:

• Is the “compilation” process of today’s AI equally deterministic and predictable? When we give it the exact same “product requirement” twice, will it generate the exact same “code”?
• Our trust in a traditional compiler is built on decades of rigorous testing, formal verification, and standardization. On what foundation should our trust in AI be built?

You suggest we should focus on “validating the product,” which seems a logical way out. But this raises a new set of questions:

• What exactly does “validation” mean here? Is it verifying that product features match the explicit list we specified, or validating the traits we never stated but that matter deeply — security, fairness, long-term maintainability, or even whether the code embeds biases we never anticipated?
• When we test a program written by a human, we largely share a common logic and world model. But AI “thinks” differently from us. Could it satisfy every test case while burying in the code “time bombs” born of its non-human logic that we could never imagine? How do we test for the things we would never think to test?

Finally, let us return to the act of “reviewing code” itself.

• Do we review code solely to find errors? Or does the process itself also entail the transfer of knowledge, the unification of standards, and a deeper understanding of the system? If we give up reviewing, do we also give up our control over the system?
• Imagine that a vastly complex, AI-built system malfunctions late one night in a bizarre, never-before-seen way. How do we diagnose and fix it when we never deeply understood its “assembly code”? Do we ask another AI to debug it? And then, what role do we play?

Your question may be revealing a future: the programmer’s role may no longer be the code “craftsman,” but the “philosopher” of requirements and the “ethicist” of system behavior. But in the transition from “craftsman” to “philosopher,” how do we ensure we don’t become the “sorcerer’s apprentice” — able only to wish upon the machine, yet utterly ignorant of its magic?

---

One thing to note is that AI productivity is growing exponentially. Once AI productivity reaches a certain magnitude, the code produced may be something humans can never finish reviewing.

The point you emphasize — the “unreviewability” brought on by exponential AI productivity growth — is the crux of the entire problem. It is a powerful constraint that forces us to stop seeking answers within the old framework and instead imagine an entirely new paradigm.

Good question. Now let us try to shift from questioner to explorer and reason out together what these answers might look like in that future of a “flood of code.”

On Trust: From “Process Trust” to “Property Trust”

Our trust in a traditional compiler is “process trust.” We trust its set of deterministic, verified transformation rules.

But in the age of AI, this will shift to Trust in Properties.

We will no longer trust AI’s process, because it is probabilistic, complex, and even a “black box.” We will trust our ability to rigorously verify the key properties of its output. This means we will develop a new generation of automated tools — which may themselves be AI — whose sole purpose is to audit and certify AI-generated code. One AI writes the code; another AI (or a formal verification system) checks whether it satisfies:

• Correctness: Does it meet the product requirements?
• Security: Are there known vulnerability patterns?
• Robustness: Does it crash under edge and exceptional inputs?
• Compliance: Does it conform to specific industry standards or regulations (e.g., GDPR)?
• Efficiency: Is resource consumption within an acceptable range?

Our trust will no longer rest on “I read the code and it’s fine,” but on “the code passed one million automated attack simulations of security scenarios and passed all formal property verifications.” The human role shifts from code reviewer to designer and final arbiter of these automated auditing systems.

On Validation: From “Testing” to “Specification” and “Adversarial Testing”

The traditional “validation” mainly means writing test cases to check that features meet expectations. After the explosion in code volume, this is far from enough.

Validation will shift its weight forward to Formal Specification and backward to Automated Adversarial Testing.

• Forward — write a “code of law,” not a “story”: The programmer’s most important work will be writing an extremely detailed, precise, unambiguous “product requirement specification.” This specification will no longer be a natural-language requirements document but more like a mathematical, logical “code of law.” AI will generate the product according to this code. Any ambiguity in the requirements may be interpreted and implemented by AI in ways we do not want. Our skill will shift from “programming” to “precisely defining the world and our expectations.”
• Backward — cultivate “natural predators”: To discover the “unknown unknowns,” we will deploy large numbers of “red-team AIs.” The sole mission of these AIs is to attack, abuse, and hunt for logic flaws in the AI product like the most devious hackers and users, trying to make it behave in violation of the “code of law.” The human role is to set the adversarial scenarios and, from the results of countless automated contests, distill the clauses of the “code of law” that need refinement.

On Review and Understanding: From “Reading Code” to “Reviewing Behavior”

We review code to understand and control the system. When the code is unreadable, we must find new ways to understand and control it.

We will give up understanding every line of code and focus instead on understanding the system’s macro-architecture and emergent behavior.

• The end of debugging, the iteration of specification: When an AI-generated system breaks, we are unlikely to single-step debug those millions of lines. Our response will be: this is a new constraint. We will describe this failure case as a new, more precise specification (“under condition X, the system must never do Y”), then have the AI regenerate a solution according to this updated “code of law.” Debugging shifts from “fixing bugs” to “refining the specification.”
• New visualization and interpretability tools: To “understand” the system, we will rely on a new generation of AI interpretability tools. They will not show us raw code but provide high-dimensional, interactive system behavior maps, logic flow diagrams, decision heatmaps. We will be like an experienced doctor reading a CT scan, not reading the molecular formula of each individual cell. We review patterns, structures, and relationships, not implementation details.
• Humanity’s final control: The human role becomes the system’s “architect” and “director of behavior.” We define boundaries, set goals, observe behavior, and revise rules. We don’t care what “syntax” the AI used to implement it; we care only that its “behavior” remains fully within our control and expectations.

Conclusion: The future you propose is highly probable. In that future, human programmers will not disappear but will undergo a profound “speciation.” We are evolving from craftsmen laying bricks one by one into “urban planners” who draft the city’s code of law, design its power and water systems, and use simulation and stress testing to ensure the city’s stability for a century.

This is undoubtedly a tremendous leap in the level of abstraction. The challenge is that we need to develop entirely new theories, tools, and ways of thinking to harness this power. We will no longer wrestle with code but with “the precision of intent” and “the complexity of behavior.” This may be a higher-order test of our species’ intellect and wisdom.