Son of AI: The Resurrection

In our age, when AI is a quick, glib, vague answer to every question about careers, the economy, and the future, it is hard to remember or even imagine a time--decades, in fact--when it was washed-up, regarded as a tried and discarded relic of the postwar era.

Once the early AI hype evaporated, relentless skeptics stepped up the attack, above all Hubert Dreyfus, who made a splash with Alchemy and AI (1965), expanded into What Machines Can’t Do (1972). Dreyfus is believed to have coined one of the most effective anti-AI metaphors: trying to reach human intelligence using AI is like trying to reach the Moon by climbing a higher tree.

At that time, rules-based programming was in ascendancy. So-called expert systems were in vogue. An example is Cyc, short for encyclopedia, one of the most famous projects of the 80s. Cyc meant to encode common-sense reality. Early AI programs didn’t “realize” some simple things, like if Abe Lincoln was in Washington in 1862, it generally meant that Lincoln’s right leg was also there. It had trouble with metaphors and fiction. It had logic quirks: “Vampires don’t exist. What is Dracula? He is a vampire. Does not compute”.

Kai-Fu Lee is an American software engineer and venture capitalist with a Taiwanese background who grew up in Tennessee. Lee’s involvement with AI dates to 1988, when he used a mathematical tool, Hidden Markov Modeling, to make a great leap forward for Apple in speech recognition. There were two “schools” of AI’s future path: rules-based and neural nets. As Lee notes, neural networks were an exciting idea, but repeated failure to live up to their promise led to their being sidelined except for limited applications.

That’s the way AI marked time. The handwriting recognition of Apple’s Newton (1993) was initially an example of half-baked, half-miraculous use of hidden Markov modeling. The persistent gap between aspiration and success was still a gaping chasm.

I first heard of neural networks, then nicknamed “connectionism,” in 1988 from a young British computer programmer. It was a strange new approach, just crazy enough to be valid. In 1994 I bought a trio of books. To paraphrase Deadpool, “Welcome to AI. You’re joining us at kind of a low point”.

Then, a dramatic breakthrough: British scientist Geoffrey Hinton’s 2006 paper on deep learning won him the Nobel Prize. “Deep learning,” a new phrase, used neural networks, focused and on steroids. It kickstarted a new wave of AI progress, climaxing in various academic and public demonstrations of the new technology in 2012 that received worldwide attention.

Hinton was really the one who sounded the starting bell on the AI wave that finally succeeded, the current one. After this point, the field accelerated so quickly it raced beyond its creators’ wildest hopes.

By then, Kai-Fu Lee had built a powerhouse career, opening Google’s offices in China as recounted in his 2018 book, AI Superpowers: China, Silicon Valley, and the New World Order. He poses the key question, “How did AI get this good, this fast?” He cites earlier technical advances—steam power, electrification, the internet—and sees several factors of today’s world, some more obvious than others, that make AI different: You can improve it almost instantly, worldwide. Second, the computing machines are much faster and cheaper than they used to be. Third, vastly more of the world’s data is accessible and machine-readable, and AI training depends on having enormous volumes of data to train on. Fourth, AI can, to some degree, write itself at greater than human speeds.

In his opinion, the final accelerant is the post-2010 rise of China in the AI field. As the title suggests, this book is an insider’s perspective on Chinese AI development from 1999 forward. As soon as AI beat a human playing Go, Asia took notice. The year 2014 seems to be particularly significant, an inflection point when China’s central government made the crucial decision to throw in, completely, with the tech revolution that was already sweeping the country.

Since then, it’s been full speed ahead, with cutthroat, street-level business competition to create champions, with Beijing’s support waiting for the winners. This often results in classic excesses, ones that capitalist economies are also prone to, overinvestment and overbuilding. But even when a state subsidy is inefficient, it can have brute force effectiveness.

Lee is especially good at coming up with specific examples of AI boosts in productivity. For example, smartphones allowed China, especially its urban residents, to leapfrog over the credit card era, going more or less directly from hiding money under a pillow to direct online payments. The AI angle comes in here: what online payment lacks is credit, and assessing credit risk is very hard for hundreds of millions of Chinese with minimal trackable history.

So gargantuan app makers do deep instant analysis of the phone owner's behavior: How quickly does he type in his birth date? Does he habitually let it run low on battery? These are things that most people would not think of or would consider irrelevant. But when taken together over millions of examples, they begin to have real predictive power.

Many of China’s traditional cultural virtues are shared with other Asian nations: respect for elders, veneration of education, thrift, and above all, hard work. But China’s nearest rival, Japan, like most of Europe and America, is a high-trust society that follows the rules and expects everyone else to. China, like Russia, is a society that is more cynical—they’d call it realistic—about behavior where money is concerned. Americans and Europeans have much higher expectations of personal privacy; the jaded Chinese (see what I did there?) always knew that their government has all their data.

Lee has a corrective note for Americans who were complacent about the challenge of Chinese AI. Yes, as with postwar Japan, China has had a long copycat period, but although intellectual theft from Westerners is still a way of life, the Chinese now have emerging AI geniuses of their own. For decades, Americans have boasted that Silicon Valley’s hippie side shows our naturally greater instinct for creativity. Lee shakes his head; Steve Jobs didn’t stack up all that loot and change the world just because he took LSD a few times and walked around barefoot. He was a ferocious capitalist, and the Chinese admire that. History is full of ironies.

The other AI books of 2016-'18 also convey a new level of progress and excitement, but Lee’s book is the only one of that bunch that already has index listings for the soon-to-be leaders Sam Altman and Demis Hassabis, about to enter our story, and though its predictions of the near future—now, basically—aren’t perfect (too optimistic about self-driving cars, too pessimistic about photorealistic image generation), they’re in the right ballpark.

Bringing us closer to the present is 2024’s Supremacy: AI, ChatGPT, and the Race that Will Change the World, by Parmy Olson, a Bloomberg journalist who covers technology. It’s a clash-of-the-titans corporate and personal conflict over the future of artificial intelligence. At the beginning of this century, few people still believed in the possibility of developing AGI, artificial general intelligence, in the foreseeable future (i.e., our lifetimes). Two very different men, 5400 miles apart, independently began to stake a claim to changing that consensus. British scientist Demis Hassabis, born 1976, started in game development, which funded his early research. Sam Altman, born 1985, entered the field through Silicon Valley venture capital.

The AI histories I read in the Nineties were laboratory and blackboard stories; by contrast, these are Wall Street stories, best at depicting personalities and the shifting power balances of corporate warfare. Altman and Hassabis follow a tradition of tech bosses putting on a pious show of concern over possible negative effects of their businesses. Increasingly, they both believed, and said publicly, that the other guy was not as scrupulous as they were. Both of them sold big stakes in their companies (Altman’s OpenAI, Hassabis’ DeepMind) to the kind of big firms they once disdained—OpenAI to Microsoft, DeepMind to Alphabet (Google).

Note to remember: if Elon Musk wants your company and you won’t sell, he gets really, really mad. He spitefully called Demis Hassabis “the Hitler of AI.”

Alas, Olson is middling at describing software development. Yes, she’s primarily writing about business, not engineering, but books like this—Richard Reeves’ The Making of the Atomic Bomb is a good example—have to be able to make analogies, to explain technical matters that will be beyond most readers without explanation. Reeves can’t make a nuclear physicist out of you, but at least you’ll understand the basics of what makes the Bomb go boom. You have to try harder than Parmy Olson does.

Okay, I’ll give it a shot. It comes to you with all the august scientific authority invested in a writer who graduated with a film school degree 53 years ago. Here goes:

The problem: Perceptrons, the original basic threshold-adjusting units of neural networks, were limited to one useful layer of “back propagation.” You could add layers, but they tended to mush together, making them useless.

Why? Think of photography. If you ever used an older Polaroid camera, you peeled the finished picture away from the negative that created it. In AI terms, the negative “trained” the positive. Similarly, the negatives made by regular cameras could be pressed directly against photo-sensitive paper: a contact print. Every point on the negative has a one-to-one correspondence with the image formed on the paper.

Now, suppose that negatives are stacked five levels deep. Would we get five useful images on paper? No, we’d lose even the one image we had; it would be a total mess. In AI, the one-to-one contact essential for training is lost.

But Geoffrey Hinton’s layers were distinct, like the pages of a book, so useful training could take place, say, between layers 3 and 4, not just between layers 1 and 2. (At the start, in 2006, Hinton went six layers deep.) A visual analogy is the way movie credit scenes are made: Mattes (special making elements) isolate the text and keep it on top of the moving image, which itself is several layers deep. They don’t blur into each other. The outermost layer is pre-trained for the specialized job of presenting output.

The next big advance in Hinton-inspired design came from the UK’s DeepMind. Parmy Olson mentions “the transformer,” a mysterious but absolutely critical stage in AI development that took place circa 2017. Olson does this without explaining it much beyond, “it can take anything and turn it into anything.”

The Transformer used multi-core processing to comprehend an entire page at a time, not just one word at a time, making it much faster. Many virtual search “heads” worked simultaneously to identify subtle hidden patterns in the numbers. It didn’t care whether the numbers represented words, pictures or chemical formulas. It was refined with the concept of “attention” in 2018.

Whether it’s text, sound, or images, the final result presented to the user isn’t formed by scanning one line at a time, like typed text or a TV image. It’s much more like seeing the picture suddenly emerge on a blank sheet of photographic paper floating in a darkroom’s developer tray. It is, in effect, apprehended at once. Like an infant’s cognitive surge, it’s a nonlinear leap in capability. The speed and detail of the response seem miraculous, especially to anyone seeing it for the first time.

Velocity has a role to play. To drive in a nail, you don’t place something heavy on the nail and push; you swing a hammer, greatly amplifying its impact. As data percolates upwards through each successive backprop layer, continuously recalculating probabilities as it goes, it picks up speed and likelihood, a kind of upwards preference cascade. Network effects help form associations with adjacent patterns. That analogy might be doing a jigsaw puzzle: when most of the pieces are already there, it’s easier to see what to solve.

OpenAI released OpenAI 3.0 to the world near the end of 2022, and it’s been a sensation, the first “miracle” application of AI that the general public became aware of. It has since been joined by Elon Musk’s Grok, Google’s Gemini, and Anthropic’s Claude, among many others.

Claude? Yep, you guessed it. Named after Claude Shannon, whose 1937 paper at MIT started all this. A field whose silicon memories are overwritten in femtoseconds sometimes has a surprising sense of history, even an indelible trace of acknowledgment and gratitude. A salute across generations. Backpropagated, of course.