When AI Becomes the Programmer: How Must Programming Languages Evolve?

When things reach their limit, they change; changing, they pass through; passing through, they endure. — Zhouyi (Book of Changes), Xi Ci II

The evolution of programming languages, as the I Ching observes, obeys the same law: exhaust, then change. When human cognition becomes the bottleneck, languages seek change through abstraction; when AI becomes the main workforce, paradigms must find passage through structure. From machine code to high-level languages, from text to AST, from implementation to specification — every transformation came about because the old order was spent and a new one opened the way. The rise of AI-native languages today is a historical inevitability, not an accident.

This essay was co-authored with Gemini. I was responsible for the outline, the narrative arc, the core arguments, and the review; Gemini’s job was to challenge my thinking and fill in the details. Thanks are also due to Cursor (Sonnet) for markdown formatting corrections and a range of title options, to gpt-5-mini for the illustration prompts, and to gpt-image-1 for the illustrations.

The Evolution of Programming Languages

Across its entire history, the story of the programming language has in essence revolved around a single core driving force: creating sensible abstractions for programmers, in service of the limits and the strengths of the human mind. Understanding this trajectory is essential if we are to grasp how fundamentally the design constraints of programming languages will shift once AI becomes the primary author of code.

The Arc of Abstraction: From Machine to Human

The history of programming languages is a history of continuously raising the level of abstraction and lowering the cognitive load on human developers. The fundamental driver is to let humans communicate with machines in a manner closer to natural thought, thereby escaping the tedium and complexity of low-level machine operations.

Programming began in its most primitive form as machine code — sequences of instructions composed of binary 0s and 1s — followed by assembly language. These “low-level languages” required programmers to think like machines, directly managing memory addresses, registers, and CPU instructions. The practice was not merely tedious and time-consuming; it was extraordinarily error-prone. In that era hardware was scarce and expensive while human labor was relatively cheap, so languages closer to the metal — the ones that could achieve higher efficiency — remained favored despite being laborious to use.

In the 1950s and 1960s, the arrival of “high-level languages” — Fortran, COBOL, and LISP foremost among them — marked a revolutionary shift. These languages introduced human-readable abstractions: symbolic notation for mathematical computation, named variables, loops, and conditionals, shielding the programmer entirely from the messy details of hardware. The fundamental goal of this leap was to make programs easier for humans to write, read, debug, and maintain.

The trend deepened over the following decades. Each new programming paradigm — structured programming as embodied in C and Pascal, object-oriented programming (OOP) as led by C++, Java, and Python, functional programming as championed by languages such as Haskell — gave human programmers more powerful tools for managing ever-growing software complexity and markedly improved development efficiency. The core payoff of all this evolution always comes down to the same thing: “abstraction and productivity.”

Analyze this historical trajectory closely and a central thesis emerges: the entire history of programming language design is a history of cognitive engineering. Every major language innovation — from the introduction of variables to the metaphor of objects to automatic garbage collection — succeeded for one fundamental reason: it lifted a concrete, tedious, and highly error-prone piece of mental labor off the human programmer and handed it to the compiler or the runtime. We no longer write assembly language as a matter of course because continuously tracking registers and memory addresses is an enormous cognitive burden for humans and a fertile source of bugs. Object-oriented programming took hold because it lets humans model complex systems using familiar “objects” with their properties and behaviors, which maps closely onto the way humans perceive the world. Likewise, automatic garbage collection became standard in modern languages because manual memory management is a vast source of stubborn defects such as memory leaks and dangling pointers — exactly the things human programmers cannot reliably get right.

A programming language’s “features,” then, do not appear out of thin air; they are targeted solutions to specific human cognitive bottlenecks. This idea lays the groundwork for understanding the coming upheaval: an AI programmer has a “cognitive” model utterly different from a human’s, and the bottlenecks it faces are utterly different too. It will therefore demand an entirely new set of language features and a design philosophy optimized for itself. Every design trade-off to date — performance versus safety, flexibility versus rigor — has been defined within the frame of human cognitive capacity. When the entity writing the code is no longer human, the baseline for those trade-offs changes completely.

Two Divergent Design Philosophies in Modern Languages: Python and Rust

In contemporary human-centered language design, two mainstream and nearly antithetical philosophies prevail. We can take two emblematic languages, Python and Rust, as the archetypes of each. They represent the two extremes of design for human developers: maximizing flexibility and productivity on the one hand, maximizing safety and correctness on the other. This philosophical dichotomy gives us an indispensable analytical framework for discussing how programming languages will evolve in the age of AI.

Python’s design philosophy places developer productivity, code readability, and ease of use above all else. Its core ideas are captured in the guiding principles known as “The Zen of Python” (PEP 20), which include such tenets as “Beautiful is better than ugly,” “Explicit is better than implicit,” and “Simple is better than complex.”

Python’s dynamic type system, automatic memory management (garbage collection), and vast, rich library ecosystem together produce its unmatched versatility and flexibility, making rapid prototyping and iteration possible. It was designed from the outset to make code “easy to learn and easy to write.” This philosophy represents the pinnacle of the historical trend toward optimizing the human developer’s immediate productivity and lowering the cognitive barrier to entry. Programmers can focus on business logic without fretting over low-level memory management or verbose type declarations. Yet this extreme flexibility has a price. The price is usually runtime performance, along with the deferral of many latent errors to runtime — errors a stricter system could have caught at compile time.

Diametrically opposed to Python, Rust’s design philosophy rests on one core premise: human programmers make mistakes, especially when dealing with complex memory management and concurrent programming. Rust’s first principle, therefore, is “safety first,” with performance and concurrency close behind.

Rust achieves this through its revolutionary ownership model and borrow checker. These mechanisms enforce strict memory safety rules at compile time, eradicating at the root entire categories of stubborn defects that have plagued systems languages like C and C++ for decades — data races, dangling pointers, double frees. The design intent is to “make the wrong thing hard to do.” It sacrifices Python-style immediate flexibility and a gentle learning curve in exchange for long-term code correctness and system reliability. Developers must satisfy the compiler’s stringent demands — a process sometimes jokingly called “fighting the borrow checker.” It may feel constraining at first, but once the code compiles, developers gain enormous confidence that the program is sound with respect to memory safety.

Rust’s philosophy represents a counter-movement: rather than pursuing unbounded freedom, it provides a powerful set of guardrails, imposing constraints at compile time to guarantee system-level correctness — a profound acknowledgment that human cognitive limits are often the source of error in complex systems programming.

To lay out the fundamental trade-off between these two philosophies clearly, the table below compares their core characteristics. It will serve as a reference point for the analysis that follows, helping us understand why existing language paradigms run into trouble when confronted with an AI programmer.

Dimension / Philosophy	Python (Human Flexibility First)	Rust (System Correctness First)
Primary goal	Developer productivity, readability, simplicity	Performance, memory safety, concurrency
Type system	Dynamic typing	Static typing with powerful inference
Error handling	Runtime exceptions (errors stay silent until execution)	Compile-time checks; mandatory handling of Result and Option
Memory management	Automatic garbage collection	Ownership, borrowing, lifetimes (managed at compile time)
Developer experience	Low barrier, rapid prototyping, gentle learning curve	Steep curve, “fighting the borrow checker,” high confidence after compilation
Core creed	“There should be one — and preferably only one — obvious way to do it.”	“Concurrency without data races, memory safety without garbage collection.”

This table lays bare the design trade-offs of the two languages. Python hands more trust and responsibility to the developer, believing they can use its flexibility to write correct code. Rust hands more trust to the compiler, using strict static analysis to guarantee the code’s low-level robustness. This fundamental philosophical difference will be magnified as never before in an era when AI is the primary programmer — and it leads directly to the need for an entirely new paradigm.

AI as Programmer: A Paradigm Shift in Code Generation

Having clearly sketched the human-centered history of programming language evolution and its core philosophical trade-offs, we now turn our gaze to the present and the future. Artificial intelligence is permeating software development at unprecedented speed and scale; it is not merely a tool upgrade but a fundamental paradigm shift. Let us dissect AI’s disruptive capabilities in code generation, and expose the inherent, decisive flaws that come with them. It is precisely these flaws that will become the driving force behind the next generation of programming languages.

The State of AI-Assisted Development

AI tools are no longer the playthings of a handful of frontier developers; they are rapidly becoming standard equipment in professional development environments, profoundly reshaping every stage of the software development life cycle (SDLC).

These AI tools display astonishing versatility, with capabilities ranging from fine-grained code completion to generating entire codebases. Over the past four or five months, more than 95% of the code in my GitHub repos has been AI-generated. I have also made many videos (see my Bilibili channel: 喜欢历史的程序君, “The History-Loving Programmer”) demonstrating the power of vibe coding from Cursor through Claude Code. These tools also provide debugging assistance and automatically generate configuration, deployment scripts, regular expressions, SQL queries, and more, greatly boosting productivity.

AI’s impact on the SDLC is far-reaching. It dramatically accelerates development, liberates developers from repetitive labor, and shortens the time from concept to market. This change is reshaping the developer’s role, turning them from “line-by-line author” of code into the “guide” and “supervisor” of AI-driven development. A new workflow paradigm is taking shape: “AI executes, human supervises.” In this mode, human developers devote more energy to higher-value activities — high-level system design, complex creative problem-solving, and the review and validation of AI-generated output.

This shift does not simply mean “writing code is faster now.” It is, in fact, the natural extension of the “arc of abstraction” discussed in Part One. Just as high-level languages freed developers from machine details, AI assistants are freeing developers from the ritual of writing routine code. Human programmers are climbing to a higher level of abstraction, their work shifting from “how to implement” to “specifying intent.” This evolution from programmer to system architect or “computational designer” foreshadows an entirely new mode of work — one that aligns with the idea of “designing intelligence environments,” in which humans build and direct a system where human and artificial intelligence collaborate. This new role will inevitably demand new tools and new languages optimized for it.

The Crisis of Correctness: The Achilles’ Heel of AI-Generated Code

For all the productivity gains AI brings, a grim reality has surfaced and now constitutes the central contradiction of the current AI programming paradigm: AI-generated code has systematic, deep-seated deficiencies in correctness, and in security above all. This “crisis of correctness” is the Achilles’ heel of today’s AI models, and a formidable challenge that existing programming languages are poorly equipped to meet.

Multiple independent academic studies and industry reports confirm the severity of the problem. One study, for example, analyzed more than 100 large language models (LLMs) and found that in 45% of cases, AI-generated code introduced security vulnerabilities. Another found that 40% of programs generated by GitHub Copilot contained defects. These are not mere syntax errors but serious security flaws lurking beneath code that looks functionally correct — high-severity vulnerabilities such as cross-site scripting (XSS, CWE-80) and log injection (CWE-117) from the OWASP Top 10. Against these specific vulnerability classes, AI’s failure-to-defend rates were as high as 86% and 88% respectively. See “Artificial-Intelligence Generated Code Considered Harmful: A Road Map for Secure and High-Quality Code Generation” https://arxiv.org/html/2409.19182v1.

The root of the defect lies in the nature of LLMs. They learn to program by training on enormous public code repositories, and that training data is itself riddled with vulnerabilities, outdated practices, and poor style. As the models learn syntax and patterns, they faithfully reproduce the defects too. Research has found that LLMs often fail to use the newer, safer features modern languages and compilers offer, gravitating instead toward obsolete or even deprecated approaches — especially in languages with heavy historical baggage like Java and C++. Moreover, AI-generated code frequently lacks the necessary defensive programming structures, leaving it more exposed to attacks such as buffer overflows or integer overflows.

More alarming still is a dangerous cognitive dissonance in the developer community. Although as many as 86% of developers express concern about AI’s security, 75% believe AI-generated code is safer than human-written code. This vast perceptual gap stems from the superficial structure and readability of AI code, which gives developers a false sense of security. Such blind trust — especially under tight project deadlines — can lead developers to accept AI suggestions without adequate review, importing serious security risks into production.

Crucially, this problem will not resolve itself as models grow larger. Research shows that bigger models do not perform significantly better on security, which suggests the problem is a systemic defect inherent in the probabilistic nature of LLMs — not a “bug” that can be fixed by simply scaling parameters. LLMs are masters of syntactic mimicry and expert pattern matchers, but they know nothing of code’s deeper semantics, logic, or security implications.

This crisis also forces us to reconsider the philosophical contest between Python and Rust. Python’s philosophy — prizing flexibility, trusting the developer, deferring error checking to runtime — is close to a disaster when the programmer is a probabilistic entity that understands no semantics. AI will happily generate code that “looks right” while violating implicit contracts, producing unpredictable runtime errors and security holes. Rust’s philosophy by contrast — enforcing strict correctness, memory safety, and thread safety at compile time — provides exactly the guardrails a non-reasoning AI agent so desperately needs.

Yet relying on an existing language like Rust is not enough either. Rust’s checks concentrate on memory and concurrency safety, whereas the errors AI introduces are all-encompassing: business logic, algorithmic correctness, and broader security protocols. This “crisis of correctness” therefore reveals a deeper need: we cannot expect “training” to make an entity that is fundamentally a probabilistic model absolutely reliable. The only viable path forward is to transform the programming paradigm itself — to move to one where correctness no longer depends on convention, testing, or human review, but is proven and enforced mathematically by the language and its toolchain. The language itself must become the final arbiter of correctness.

The Morphology of an AI-Native Programming Language

Synthesizing the two preceding parts — the history of programming languages evolving to serve human cognition, and the profound “crisis of correctness” introduced by AI as programmer — we can begin to sketch a blueprint for a future paradigm. This paradigm is not designed for humans to write directly, but optimized for AI to generate code efficiently and safely under high-level human supervision. It will no longer be a “programming language” in the sense we know today, but an entirely new, multi-layered intelligent system. This part describes in detail the shape and core characteristics such an AI-native language might take.

Beyond Text Files: From Syntax to Structure

The programming language as we understand it today takes as its most basic form a sequence of characters stored in a text file. This text-centric paradigm evolved to suit human reading and writing habits. For an AI, however, this linear, ambiguity-laden textual representation is an inefficient and unnatural medium. The core of future AI-native programming will shift from manipulating “syntax” to operating directly on “structure.”

Making the Abstract Syntax Tree (AST) the Single Source of Truth

Future programming environments will be built on projectional editing (also called structural editing). In this model, a program’s canonical representation is no longer its textual code but its abstract syntax tree (AST). The text we see is merely one configurable “projection” or “view” of that tree.

An AI agent will no longer write code by “typing” characters; it will construct and modify AST nodes directly in memory. To create a variable declaration, for instance, the AI instantiates a VariableDeclaration node and populates its child nodes — name, type, initializer. For a machine, this is far more direct, efficient, and unambiguous than parsing a string of characters. It eliminates at the root all the syntactic ambiguity associated with parsing, which is essential to achieving safe language composition.

In this paradigm, human developers will interact with the program’s AST through a highly intelligent IDE. The IDE can project the same AST into many different views as the task demands: the traditional text view, a graphical view for data-flow analysis, a tabular view for configuration, and so on.

Making the Intermediate Representation (IR) the Lingua Franca

Going further, a well-defined, machine-oriented intermediate representation (IR) may become the true lingua franca of the AI-driven development era. Compilers have long used IRs as the bridge between front end (parsing a high-level language) and back end (generating target machine code).

Compared with the AST of a high-level language, an IR is typically simpler, more regular, and closer to low-level operations. It is designed to be language- and platform-agnostic, which makes it a perfect universal format to be generated, analyzed, and manipulated by different AI models. AI can perform program transformation, optimization, and reasoning at this more fundamental level without caring about the “syntactic sugar” that upper-level languages designed for humans.

This shift from text to structure solves a key difficulty that projectional editing has historically faced. In the past, a major obstacle was the lack of a widely accepted, non-textual persistence format, along with a supporting tool ecosystem (version control, diff tools) that is taken for granted in a world of text files. In an AI-dominated development environment, however, the problem is neatly inverted. AI is perfectly capable of handling structured storage formats — XML, or more efficient binary tree representations. A new generation of AI-native IDEs will implement version control and diffing at the structural level (AST/IR) and project structural diffs back out in human-readable form. The problem of non-textual source files is, at bottom, merely a problem of an old, human-centered workflow.

The ultimate consequence of this shift is the death of syntax. Endless arguments about code style — “tabs or spaces,” “braces on their own line or not” — become meaningless. Syntax degrades entirely into a configurable “theme” or “skin” that a user can switch at will in the IDE. Developer A can view a program in a Python-like, brace-free style, while her colleague Developer B views the exact same underlying AST projected with C-style braces. A language’s essence is its structure, not its textual appearance. This is the ultimate expression of the principle of separation of concerns.

Verification at the Core: Built on a Foundation of Proof

To address the “crisis of correctness” raised in Part Two at its root, the core of an AI-native language must be founded on the principles of formal verification and provability. It must push Rust’s philosophy of guaranteeing correctness through compile-time checks to its logical extreme.

The most powerful weapon for solving the problem of AI code correctness is to make the language’s type system itself a proof tool. We have already seen the power of this in Rust: a well-designed type system can even guarantee concurrency safety, something other languages find very difficult to achieve.

Beyond that, we can employ formal verification — using mathematical methods to prove that a program strictly obeys its specification. Historically, formal verification has rarely been used in large commercial software projects because of its high cost and steep expertise barrier. In an AI-native world, that situation will be completely transformed.

AI has the potential to automate much of the drudgery in the formal verification pipeline. The human supervisor’s role will become defining the system’s high-level properties and invariants — its “specification” — while AI takes on the daunting task of generating the underlying implementation and its formal proof, using integrated proof assistants and SMT solvers. The development process will evolve into a continuous integration loop of “specification definition, code-and-proof generation, automated verification.”

This marks the birth of an entirely new computational contract. In this contract, humans and AI have a clear division of labor that plays to each side’s strengths:

• Human: provide the specification. This includes defining what the system “should do,” expressing intent precisely through high-level properties, invariants, and dependent types.
• AI: provide the implementation and its proof. This includes generating the low-level code that satisfies the specification (in IR/AST form) while simultaneously constructing a formal proof verifiable by type checkers and verification tools.

This division of labor resembles the collaboration between an “architect/product team” and a “development and QA team” in an effective software organization — except the latter role is now played efficiently and reliably by AI. This is the ultimate expression of extending Rust’s “make errors hard to commit” philosophy from memory and concurrency safety to every logical layer of the program. It replaces runtime tests, which can only demonstrate the presence of particular errors, with compile-time proofs, thereby guaranteeing correctness in theory for all possible inputs.

The Human Interface: From Prompt Engineering to Intelligence Design

Under the AI-native paradigm, the human role and mode of interaction will also change fundamentally. Humans will no longer be artisans of code but designers and commanders of intelligent systems.

Human work will extend well beyond today’s “prompt engineering.” Borrowing Stephen Wolfram’s conception, humans will use a high-level “linguistic interface” to convey intent to a deterministic “computational bedrock.” This interaction will not be one-way instruction-giving but an iterative dialogue of clarification and refinement, aimed at jointly perfecting the system’s specification.

The human role will evolve into “designing intelligence environments,” into genuine “system architects.” Their core responsibility will be to define the system’s goals, constraints, business rules, and desired properties, delegating implementation detail entirely to AI. Human creativity will concentrate on higher-order problem-solving and innovation.

For humans to effectively supervise and trust complex AI-generated systems, transparency is essential. Future programming “environments” must therefore treat explainable AI (XAI) as a first-class core feature.

When AI generates a complex algorithm or a formal proof, it must simultaneously be able to produce an explanation a human can understand. Such explanations can take many forms: a natural-language description summarizing its design decisions, a visualization showing data flow and logical dependencies, or a step-by-step walkthrough of its proof logic. This explainability is indispensable for debugging, auditing, security review, and ensuring AI behavior stays aligned with genuine human intent.

In this new paradigm, the most valuable human skill will no longer be concrete coding technique but epistemology — the theory of knowledge. The highest duty of the human programmer will be to define, precisely and unambiguously, what counts as “truth” in the system they want to build. They must become experts at articulating requirements, constraints, and invariants. This marks programming’s transformation from a technical craft into an intellectual activity closer to logic and philosophy. Put simply: humans define the axioms, and AI derives the theorems — which are the programs.

Conclusion: The Emergence of a Dual-Layer Intelligence Environment

In sum, as the primary workforce of programming shifts from humans to AI, the evolution of programming languages will transcend improvements in syntax and move toward an entirely new architecture: the Dual-Layer Intelligence Environment.

This architecture does not seek a compromise between the “human-friendly flexibility” of Python and the “machine-enforced correctness” of Rust. Instead, it dissolves this historical contradiction through a thorough separation of concerns.

• The AI core layer (computational bedrock): a representation layer optimized for machines and founded on verifiability. Structurally it is structured (IR/AST-based); theoretically it rests on dependent types and formal methods; its design goal is absolute logical rigor and provable correctness. It pushes Rust’s philosophy to its extreme, constructing a computational world that AI can operate in safely and efficiently — a world that may be alien, even incomprehensible, to direct human cognition.
• The human shell layer (interaction interface): an interaction layer designed for humans, flexible and multimodal. In form it is conversational, explainable, and projectional. It allows humans to express intent, define specifications, and understand AI behavior through high-level abstractions — natural language, diagrams, formulas. This layer embodies the essence of Python’s philosophy — human-centered, prizing ease of expression and the free flow of thought — except that the level it operates on has risen from “implementation” to “specification.”

Programming in the future will no longer be about inventing a better syntax but about building a better partnership. That partnership is supported by an entirely new architecture, one that supplies the right “language” to the right “programmer,” whether human or AI. Humans command through expressive, flexible dialogue; AI executes within a world of logical certainty and verifiability. That is the ultimate picture of how programming languages will evolve in the age of AI.