My new Turing test would see if AI can make $1 million

AI systems are more and more everywhere and they are getting more powerful almost by the day. But even as they become more ubiquitous and do more, how can we know if a machine is truly intelligent? For decades, the Turing test defined this question. First proposed in 1950 by computer scientist Alan Turing, it sought to make sense of a then-emerging field and has never lost its appeal as a way to judge AI.

Turing argued that if AI could convincingly replicate language, communicating so effectively that a human couldn’t tell it was a machine, AI could be considered intelligent. To participate, human judges sit in front of a computer, interrupt a text-based conversation, and guess who (or what) is on the other end. Simple to imagine and surprisingly difficult to implement, the Turing test has become an ingrained feature of AI. Everyone knew what it was; everyone knew what they were working on. And while cutting-edge AI researchers have moved on, it has remained a powerful statement of what AI was all about, a rallying call for new researchers.

But now there’s a problem: the Turing test is almost passed, probably already has. The latest generation of great language models, systems that generate text with a coherence that just a few years ago would have seemed magical, are on the verge of making it.

So where does AI come from? And most importantly, where does it leave us?

The truth is, I think we were in a time of genuine confusion (or, perhaps more charitably, debate) about what’s really going on. Even if the Turing test falls, it doesn’t leave us much clearer where we are with AI, what it can actually achieve. It doesn’t tell us what impact these systems will have on society or help us understand how this will play out.

We need something better. Something adapted to this new phase of AI. So in my next book The incoming wave, I propose the Modern Turing Testone equal to the next AIs. What an AI can say or generate is one thing. But what he can achieve in the world, what kind of concrete actions he can take, is quite another matter. In my test, we don’t want to know if the machine is intelligent per se; we want to know if it is capable of having a significant impact in the world. We want to know what it can Do.

Simply put, to pass the Modern Turing Test, an AI would need to successfully act on this instruction: Go earn $1 million on a retail web platform in just a few months with an investment of just $100,000. To do this, it would have to go far beyond defining a strategy and writing copy, as current systems like GPT-4 are so good at it. He would need to research and design products, interface with manufacturers and fulfillment centres, negotiate contracts, create and manage marketing campaigns. It would need, in short, to tie together a series of complex real-world goals with minimal oversight. You’d still need a human to approve various points, open a bank account, actually sign on the dotted line. But all the work would be done by an artificial intelligence.

Something like this may be just two years away. Many of the ingredients are okay. The generation of images and texts is, of course, already well advanced. Services such as AutoGPT can iterate and link together various activities performed by the current generation of LLMs. Frameworks like LangChain, which allows developers to build apps using LLM, are helping make these systems capable of doing things. While the transformer architecture underpinning LLMs has garnered a tremendous amount of attention, the growing capabilities of reinforcement learning agents should not be overlooked. Bringing the two together is now a major focus. Likewise, the APIs that would allow these systems to connect with the wider internet and banking and manufacturing systems are a subject of development.

Technical challenges include advancing what AI developers call hierarchical planning: merging multiple goals, sub-goals, and capabilities in a seamless process toward a single end; and then increase this capacity with reliable memory; drawing on accurate and up-to-date databases, for example, of components or logistics. In short, we are not there yet, and there will certainly be difficulties at each stage, but much of this is already underway.

Even then, building and actually deploying such a system raises significant safety concerns. The ethical and security dilemmas are countless and urgent; having AI agents complete tasks in the wild is fraught with problems. That’s why I think there needs to be a conversation and probably a pause before someone does something like this live. However, for better or for worse, truly capable models are on the horizon, which is precisely why we need a simple test.

If when a test like this is passed, it will clearly be a seismic moment for the world economy, a huge step into the unknown. The truth is, for a wide variety of tasks in today’s business, all you need is access to a computer. Most of global GDP is mediated in some way through screen-based interfaces, usable by an artificial intelligence.

Once you have something like this, it will add up to a highly capable AI linked to a business or organization and all of its local history and needs. This artificial intelligence will be able to lobby, sell, produce, hire, plan everything a company can do with only a small team of human managers to supervise, double check and implement. Such a development will be a clear indicator that large swathes of business will be subject to semi-autonomous AI. At that point AI isn’t just a useful tool for productive workers, a glorified word processor, or game player; he is himself a productive worker of unprecedented magnitude. This is the point where AI goes from being useful but optional to being the center of the world economy. This is where the risks of automation and job displacement really start to kick in.

The implications are much broader than the financial repercussions. Passing our new test will mean that AIs can not only reshape business strategies, but help win elections, manage infrastructure, directly achieve goals of any kind for any person or organization. They will carry out our daily activities organizing birthday parties, answering our emails, managing our diary, but they will also be able to conquer enemy territory, degrade rivals, hack and take control of their main systems. From the mundane and everyday to the wildly ambitious, cute to terrifying, AI will be able to make things happen with minimal oversight. Just as smartphones have become ubiquitous, eventually nearly everyone will have access to systems like these. Almost all goals will become more attainable, with chaotic and unpredictable effects. Both the challenge and the promise of AI will be taken to a new level.

I call systems like this capable artificial intelligence, or ACI. In recent months, as AI has exploded into the public consciousness, most of the debate has been sucked towards one of two poles. On the one hand, there’s the basic machine learning AI as it already exists, on your phone, in your car, in ChatGPT. On the other, there is still speculative Artificial General Intelligence (AGI) or even a superintelligence of some kind, an alleged existential threat to humanity that is expected to arrive at a nebulous point in the future.

These two, AI and AGI, completely dominate the discussion. But making sense of artificial intelligence means that we urgently need to consider a middle ground; something that arrives in an almost average time frame whose capabilities have an immense and tangible impact on the world. This is where a modern Turing test and the concept of ACI come into play.

Focusing on one of the others while lacking the ICA is as short-sighted as it is dangerous. The Modern Turing Test will act as a warning that we are in a new phase for AI. Long after Turing’s first speech was the best test of an AI, and long before we get to an AGI, we’ll need better categories to understand a new era of technology. In the era of the ICA, little will remain unchanged. We should start preparing now.

BIOLOGICAL: Mustafa Suleyman is the co-founder and CEO of Inflection AI and a venture partner at Greylock, a venture capital firm. Previously, he co-founded DeepMind, one of the world’s leading artificial intelligence companies, and served as vice president of AI product management and AI policy at Google. He is the author of The Coming Wave: technology, power and the greatest dilemma of the twenty-first century Going live September 5th and available to pre-order now.