Header image: John William Waterhouse, Public domain, via Wikimedia Commons
A lot of people smarter than me have been sounding the alarm about the potential threat that AI poses for some time, and I have chosen to ignore it. Until recently, I had never had a conversation with an AI that sounded even halfway plausible. Demos are fabrications, and until I encountered something in my own experience that gave me pause, then the threat was always somewhat theoretical. Alien invasion, not nuclear war.
Now, the Rubicon that was the Turing test has not only been crossed, but the river itself has had a suspension bridge built over it. Even though the primary use of the technology at this point is to produce a limitless quantity of bullshit the potential impacts are immediate, ranging from education, to hiring, to human relationships (a friend told me that they use ChatGPT to generate answers to reply to putative partners on dating apps).
I have worked my entire life in disparate fields of science and technology, and so I have a general optimism that advancement betters us. Machine learning has given us AlphaFold, new mechanisms of drug discovery, and the potential to tame the firehose of the scientific literature. So let's be very careful that when we talk about the threat of AI that we acknowledge that the potential benefits are well beyond what we can ethically discard.
Twenty years ago, the great ethical debate in science was over biotechnology, particularly pertaining to designer babies and engineered pathogens. This seems to have an echo in the present moment for two reasons: (1) the difficulty of reconciling definite benefits with the possibility of catastrophe, and (2) the queasiness we feel about pushing the limits of what is or is not a person. The key assumption seemed to be that once we had crossed a certain threshold, there was no going back. By 2000 moving a single gene from one organism to another was a simple matter, and we were on the verge of having access to entire genomes. Moreover, the number of genes in the human genome appeared to be a lot less that anyone had thought (in the first years after the genome was completed, it was thought to be about 30,000). Put those two advances together and life itself was our Play-dough, with all the possibility and terror that entailed.
Only: it kind of never happened. There are no designer babies and we don't appear to be any closer to having them than we were 20 years ago (I'll leave aside the question of whether we have engineered pathogens for another day). The reason, I think, is emergence. Engineering a single gene is a incredibly simple compared to engineering a genome, because if we have 30 thousand genes then we have 900 million pairwise combinations of genes and 810 trillion three-way combinations. The effects of changing a lot of the genome at the same time is pretty hard to reason about: natural selection does it by trial and error, with a luxury of time we don't have.
This, ironically, is probably my greatest caution about AI. Engineered systems are put together in different ways than natural systems: the various components are connected through small interfaces that can be reasoned about. Evolution does not have this constraint because it works by trial and error. This makes bio-engineering hard. On the flip side: the apparent success of models like ChatGPT discard the engineering model and let the programs build themselves. Nobody really understands how they work. This is probably related to the "hallucinations" commonly described in language models, the weaknesses in Alpha Go, and the continuing problems with self-driving cars. The flaws seem obvious to an intelligent human; but with no bridge between human and machine intelligence, they are not fixable (except for bigger training sets, larger models and therefore more complexity).
One self-styled "modern-day polymath" predicts that coding won't exist in five years. I don't know whether to hope he's hopelessly naive or resign myself to using the bullshit machine. Should we use computers, trained on human-written code, to program other computers? At worst, we are at risk of releasing a lot of code into that world that no one wrote and no one understands. At best, hiring developers is about to get a whole lot harder than it already is.
I see the immediate danger not as AI replacing programmers, but as managers, stakeholders, users, designers, thinking that AI can replace coders. ChatGPT is admittedly good, scary good, at solving general problems with little domain specific input, like FizzBuzz or creating a login form in React. We use these simple problems to as a sort of town square for our craft: publishing tutorials, demonstrating a new framework or library, or interviewing developers. But they bear the same resemblance to the problems faced by a working software engineer as a five-paragraph essay on Lord of the Flies bears to a graduate thesis. The aim of any code is to solve a unique problem, considering written requirements, mockups, knowledge of the current software and knowledge of how it will change in the future, and domain-specific knowledge of the technological foundations. The more specific a problem, the greater the capability for ChatGPT to hallucinate. The more general a problem, the less need there is to write any code at all.
The responsible use of any technology depends on knowing the limitations of that technology. The text, code, and answers generated by ChatGPT are best thought of as bullshit: not because they are always wrong (they are usually right), but because we have no way to interrogate their veracity. Unlike search engines, the de facto oracle of our civilization for the last 20 years, it does not say (indeed, it cannot) how it knows what it knows. It does not link back to its source material. If you ask it for a list of references, it makes them up. If you tell it it's wrong, it always agrees with you. There is a person behind the curtain pulling the levers, but the person has been averaged with a thousand other people and anonymized (without their explicit consent). From the perspective of its creators, this anonymization is a Feature Not A Bug, as it presents the machine as an oracle. It knows nothing, yet presents as being omnipotent, and this omnipotence is mysticism, not science.
How much damage has already been done by the very human confusion of artificial intelligence with divine intellect? How many convicts have been denied parole, how many job candidates have been passed over, by modern-day phrenology? How many ISIS fighters and QAnon advocates have been recruited by recommendation algorithms? I am not advocating smashing the servers. I am advocating responsibility. A human which generated bullshit as thoughtlessly as ChatGPT would quickly lose the trust of their social circle. The problem, at least for now, is not that they will replace us, but that they will do a convincing job of making us believe that they can.