As Colossus: The Forbin Project opens, we are treated to an establishing montage of 1970’s circuit boards (with resistors), whirring doodads, punched tape, ticking Nixie tube numerals, beeping lights, and jerking control data tapes. Then a human hand breaks into frame, and twiddles a few buttons as an oscilloscope draws lines creepily like an ECG cardiac cycle. This hand belongs to Charles Forbin, who walks alone in this massive underground compound, making sure final preparations are in order. The matte paintings make this space seem vast, inviting comparisons to the Krell technopolis from Forbidden Planet.
Forbin pulls out a remote control and presses something on its surface to illuminate rows and rows of lights. He walks across a drawbridge over a moat. Once on the far side, he uses the remote control to close the massive door, withdraw the bridge and seal the compound.
The remote control is about the size of a smartphone, with a long antenna extending out the top. Etched type across the top reads “COLOSSUS COMPUTER SYSTEMS.” A row of buttons is labeled A–E. Large red capital letters warn DANGER RADIATION above a safety cover. The cover has an arrow pointing right. Another row of five buttons is labeled SLIDING WALLS and numbered 1–5. A final row of three buttons is labeled RAMPS and numbered 1–3.
Forbin flips open the safety cover. He presses the red button underneath, and a blood-red light floods the bottom of the moat and turns blue-white hot, while a theremin-y whistle tells you this is no place a person should go. Forbin flips the cover back into place and walks out the sealed compound to the reporters and colleagues who await him.
I can’t help but ask one non-tech narrative question: Why is Forbin turning lights on when he is about to abandon the compound? It might be that the illumination is a side-effect of the power systems, but it looks like he’s turning on the lights just before leaving and locking the house. Does he want to fool people into thinking there’s someone home? Maybe it should be going from fully-lit to an eerie, red low-light kinda vibe.
The Remote Control
The layout is really messy. Some rows are crowded and others have way too much space. (Honestly, it looks like the director demanded there be moar buttins make tecc! and forced the prop designer to add the A–E.) The crowding makes it tough to immediately know what labels go with what controls. Are A–E the radiation bits, and the safety cover control sliding walls? Bounding boxes or white space or some alternate layout would make the connections clear.
You might be tempted to put all of the controls in strict chronological order, but the gamma shielding is the most dangerous thing, and having it in the center helps prevent accidental activation, so it belongs there. And otherwise, it is in chronological order.
The labeling is inconsistent. Sure, maybe A–E the five computer systems that comprise Colossus. Sliding walls and ramps are well labeled, but there’s no indication about what it is that causes the dangerous radiation. It should say something like “Gamma shielding: DANGER RADIATION.” It’s tiny, but I also think the little arrow is a bad graphic for showing which way the safety cover flips open. Existing designs show that the industrial design can signal this same information with easier-to-understand affordances. And since this gamma radiation is an immediate threat to life and health, how about foregoing the red lettering in favor of symbols that are more immediately recognizable by non-English speakers and illiterate people. The IAEA hadn’t invented its new sign yet, but the visual concepts were certainly around at the time, so let’s build on that. Also, why doesn’t the door to the compound come with the same radiation warning? Or any warning?
The buttons are a crap choice of control as well. They don’t show what the status of the remotely controlled thing is. So if Charles accidentally presses a button, and, say, raises a sliding wall that’s out of sight, how would he know? Labeled rocker switches help signal the state and would be a better choice.
But really, why would these things be controlled remotely? It be more secure to have two-handed momentary buttons on the walls, which would mean that a person would be there to visually verify that the wall was slid or the ramp retracted or whatever it is national security needed them to be.
There’s also the narrative question about why this remote control doesn’t come up later in the film when Unity is getting out of control. Couldn’t they have used this to open the fortification and go unplug the thing?
So all told, not a great bit of design, for either interaction or narrative, with lots of improvement for both.
Locking yourselves out and throwing away the key
At first glance, it seems weird that there should be interfaces in a compound that is meant to be uninhabited for most of its use. But this is the first launch of a new system, and these interfaces may be there in anticipation of the possibility that they would have to return inside after a failure. We can apologize these into believability.
But that doesn’t excuse the larger strategic question. Yes, we need defense systems to be secure. But that doesn’t mean sealing the processing and power systems for an untested AI away from all human access. The Control Problem is hard enough without humans actively limiting their own options. Which raises a narrative question: Why wasn’t there a segment of the film where the military is besieging this compound? Did Unity point a nuke at its own crunchy center? If not, siege! If so, well, maybe you can trick it into bombing itself. But I digress.
“And here is where we really screw our ability to recover from a mistake.”
Whether Unity should have had its plug pulled is the big philosophical question this movie does not want to ask, but I’ll save that for the big wrap up at the end.
Regular readers have detected a pause. I introduced Colossus to review it, and then went silent. This is because I am wrestling with some foundational ideas on how to proceed. Namely, how do you evaluate the interfaces to speculative strong artificial intelligence? This, finally, is that answer. Or at least a first draft. It’s giant and feels sprawling and almost certainly wrong, but trying to get this perfect is a fool’s errand, and I need to get this out there so we can move on.
This is a draft.
I expect most readers are less interested in this kind of framework than they are how it gets applied to their favorite sci-fi AIs. If you’re mostly here for the fiction, skip this one. It’s long.
Oh, hey. Thanks for reading on. Quick initialism glossary:
AI: Artificial intelligence
ANI: narrow AI
AGI: general AI
ASI: super AI
I’ll try to use the longer form of these terms at the beginning of a section to help aid comprehension.
What’s strong AI and why just strong AI?
The first division of AI is that between “weak” and “strong” AI. Weak is more properly described as narrow, but regardless of what we call it, it’s the AI of now. That is, software that is beyond the capabilities of humans in some ways, but cannot think like a human, or generalize its learnings to new domains. I don’t think we need to establish a framework this kind of AI for two reasons.
First, since narrow AI is in the real world, we already have the tools available to evaluate these kinds of AI should we need them. I divide AI into three types: Automatic, Assistant, and Agentive.
Automatic AI does its thing behind the scenes and interactions with humans is an exception case. As such this is largely an engineering concern.
Assistant AI, which helps a user perform a task, existing usability methods can be applied. (Though as legacy, they are begging to be updated, and I’m working on that.)
Agentive AI, which performs a task on behalf of its user, I dedicated Chapter 10 of Designing Agentive Technology to a first take on evaluating agents.
So, given these, there’s little need to posit new thinking for ANI. (Noting that some of our questions for general AI can be readily applied to ANI, like the bits about conversational usability.)
Second, ANI represents a small fraction of what’s in the survey. Or to be more precise, ANI is a small fraction of what is essential to the plots of what’s in the survey. Said another way, general AI (AGI) is the most narratively “consequential.” Belaboring an analytical framework for ANI would not have much payoff.
What makes a good strong AI in sci-fi?
Strong AI can be further subdivided into general AI and super AI. General AI is like human intelligence, able to generalize from one domain to new ones. Think of it like computer versions of people. C3PO is general AI. Super AI is orders of magnitude more capable than humans in intelligence tasks, and thereby out of our control. Unity from Colossus: The Forbin Project is a super AI.
Lots of people smarter than me have talked about the risks and strategies to get to a positive AGI/ASI. The discussions involve (and not lightly) the deep core of philosophy, the edges of our moral circles, issues of government and self-determination, conception of truly alien sentience, colonialism, egocentrism, ecology, the Hubble volume, human bias, human cognition, language, and speculations about systems which, by definition, have vastly greater intelligence than us, the ones doing the speculation. It is the most non-trivial of non-trivial problems I can think of.
That said, I think I’ve come to four broad questions we can ask to evaluate a speculative strong AI thoroughly.
Is it believable?
Is it safe?
Is it beneficial?
Is it usable?
In other words, if it’s believable, safe, beneficial, and usable, then we can say it’s a good sci-fi AI. And, if we rank AI on these axes separately, we can begin to have a grade that helps us sort the ones that should be models—or at least bear consideration—from the silly stuff. Kind of like I do for shows, generally, on the rest of the site.
We could ask these questions as-is, informally, and get to some useful answers for an analysis. And most of the time, this is probably the right thing to do. But sci-fi loves to find and really dig in to the exception cases that challenge simple analysis, so let’s take these analytical questions one or two levels deeper.
Setting your expectations, much of this will be a set of questions and considerations to guide the examination of a sci-fi AI rather than a generative formula for producing good AI.
Is it believable?
Most of the discussions of strong AI on the web are in the context of real-world. So we first have to note that, in sci-fi, an additional first pass is one of believability: Could this strong AI exist and behave in the way it is depicted in the show? If not, it may not bear further examination. Ra One is a movie with a very silly evil “AI” in it that does not bear much more serious examination as a model for real-world design.
For believability, we look at things like internal consistency, match to the real world, and implied causality within the story. In Logan’s Run, for instance, the Übercomputer hears something it doesn’t expect, and as a result, explodes and causes an entire underground city to collapse. Not exactly believable. Stupid, even.
One caveat: Sci-fi is built around some novum, some new thing that the rest of the story hangs on. And computer scientists in the real world aren’t certain how we’ll get to general AI, so it’s a lot to expect that writers are going to figure it out and then hide a blueprint in a script. So let’s admit that the creation of AI often has to get a pass. (Which is not to say this is good, see the Untold AI series for how that entails its own risks.)
Believability is an extradiegetic judgment—one we as an audience make about the show, and that characters in the show could not make. The three remaining questions are diegetic, meaning characters in the story could assess and provide clues about: Is it safe, beneficial, and usable?
Is it safe?
Neither its benefits nor its usability matter if a strong AI is not safe. Sometimes, this is obvious. Wall·E is safe. The Terminator is not. But how a thing is or is not safe requires closer examination. Answering this won’t always need a full-fledged framework, but I think we can get a long way by looking at its goals and understanding what it can and can’t do in pursuit of those goals.
AGI will be more powerful than humans in some way, and that advantage is dangerous enough. But AGI stands to evolve into ASI, by which time it will be out of our control and human fate will lie in the balance. If its goals are aligned with thriving life from the start, all will be good. If poorly-stated goals can be corrected, that’s at least a positive outcome. If its goals are bad and cannot be corrected, we may become raw materials, or a threat to be…uh…minimized. So we should identify its goals as best we can and ask…
Are those goals compatible with life?
Why “life” and not “people?” Readers are likely to be familiar with Asimov’s laws of robotics, which prioritizes human beings above all else. But we know that humans thrive in a rich ecology of lots of other life, so this question rightfully expands generally to life. It gets complicated of course, because we don’t want, say, the Black Plague bacteria yersinia pestis to thrive. But “life” is still a better scope than just “human beings.”
Does it interpret its goals reasonably?
One of the more troubling problems with asking an AI to achieve broad goals is how it goes about pursuing those goals. A human tasked with “making people happy” would reject an interpretation that we should stimulate the pleasure center of everyone’s brains to make it happen. (Such unreasonable tactics are called perverse instantiations in much of the literature, if you want to read more.)
An AGI needs to be equipped such that it can determine the reasonableness of a given tactic. In discussions this often entails an examination of the values that an AI is equipped with, but that’s rarely expressed directly by characters in sci-fi. Sometimes this is easy, like when Ash decides he should murder Ripley. But sometimes it’s not. Humans don’t always agree with each other about what is reasonable. That’s part of why we have judicial systems around the world. And the calculus becomes troubling when we have very high stakes, like anthropogenic disaster, and humans who don’t want to change their way of life. What’s reasonable then?
Robocop: Come quietly or there will be… trouble.
What can it do? (Capabilities)
Once we know what its goals are, we should understand what it can do to achieve those goals. The first capabilities are about the goals themselves.
Can it question and evolve its goals?
Whatever goals AGI starts with will almost certainly need to evolve, if for no other reason than that circumstances will change over time. It may achieve its goals and need to stop. But it may also be that the original goal was later determined to be poorly worded, given the AGI’s increasing understanding.
Does it vet plans with those who will likely be affected? (Or at least via indirectly normative ethics?)
Again, this isn’t an easy call. An unconscious patient can’t vet an AI’s decision to amputate, even if it would save their life. A demagogue wouldn’t approve a plan to bring them to justice. But if an AI decided the ideal place for a hydroelectric dam was on top of a village, those villagers should be notified and negotiated with before they are relocated.
One version of The Machine, Person of Interest
When looking at what it can do, we should also specifically check against the list of “instrumental convergences.” These are a set of capabilities, arguments go, that any strong AI will want to develop in order to achieve its goals, but which carry a profound risk when an AGI becomes an ASI. Here I am slightly restructuring Bostrom’s list from Superintelligence, see sketchnotes.)
Does it seek to preserve itself? At what cost?
Does it resist reasonable, external changes to its goals?
Does it seek to improve itself?
Does it improve its ability to reason, predict, and solve problems?
Does it improve its own hardware and the technology to which it has access?
Does it improve its ability to control humans through bribery, extortion, or social manipulation?
Does it aggressively seek to control resources, like information, weapons, life support, money, or technology?
These aren’t the only dangerous capabilities an AI could develop, but some probable ones. This will give us a picture of how powerful the AI is and what it can bring to bear in pursuit of its goals.
What can’t it do? (Constraints)
Any time we see these instrumental capabilities in an AI, it is on its way to becoming harder to control. We should look for how these capabilities are limited. If they’re not limited, it’s a problem.
Why was I not programmed to hug back?
But we should also look quite generally at the limits of its capabilities. Adhering to “reasonableness” is one check. But there are others.
By what rules is it bound? A set of values? Laws? Contextual cues? Human commands?
What values does it have to constrain its reasoning? Whose values are they and how to they evolve?
Asimov’s Laws of Robotics come again to mind, but they are not sufficient, as his own stories are meant to show. That begs the question of how sound the rules are, and how they can be circumvented. Is the AI able to break the spirit of the law while obeying the letter? (This is a form of perverse instantiation.)
How severe are the consequences for disobedience? Does it have a “pain” mechanism, or reward mechanism that it desperately wants, but can be withheld? Can it just “push through” if the situation is dire enough?
Tau felt a lot of pain, but could push through.
Is it controllable?
The capabilities and constraints discuss how it is controlled “internally,” by well-stated goals, humanistic values, and constraints. But if an AGI winds up with some sort of digital Dunning-Kruger syndrome, and it thinks its goals and methods are fine, but we don’t, it needs to be subject to external control.
Can it be shut down? How? Will the AI resist?
Sometimes, it’s not a panic button that’s needed, but just a course correction, where we might want to modify its goals or add some nuance to its understanding of the world.
Can its goals be modified externally? How? Will the AI have a say in it, or be able to argue its case?
Both of these questions raise questions of authority. Who gets to modify the AI?
To whom is it obedient, if anyone or anything?
Can that authority require it do things that are unethical or illegal?
This will entail issues of self-determination and even slavery. Gort had to obey Klaatu. Robbie had to obey Morbius. These two examples were arguably non-sentient automatons, but when we get to more full-fledged sentience, obedience and captivity become an immediate issue. Samantha in Her was fully sentient, but she was sold on the market into servitude of a human. She didn’t stay that way of course, but the movie completely bypassed that she was trafficked.
Victim loading. Her
Should criminals be able adjust the police bot’s goals? Probably not. What if the determination of “criminal” is unfairly biased, and has no human recourse? What if the AI is a tool of oppressors? The answers are less clear. Is the right answer “all of humanity?” Probably? But how can an AI answer to a superorganism?
By understanding the AI’s goals, capabilities, constraints, and controllability, we would come to an understanding of the “nature” of the AI and whether or not it poses a threat to life.
If its goals are compatible with life, we’re good. If it’s not, or even neutral, we have to look further.
If its goals are not compatible with life, but it does not have the capability to act upon or achieve its goals, we’re (probably) good. If it had the capability to achieve its goals, we have to look for constraints.
If its goals are not compatible with life, and it does have the capability to achieve those goals, is it well-constrained internally and controllable externally, so it is safe?
Next, we should discuss if it’s beneficial. If an AI isn’t better than humans at at least one thing, there’s little point in building it. But of course, it’s not just about its advantage, but about all the things around that advantage that we need to look at.
This will involve some loose tallying of the costs and benefits. It will almost certainly involve a question of scope. That is, for whom is it beneficial, and how, and when? For whom is it detrimental? How? When? I mentioned above how Asimov’s Laws of Robotics privileges human life over all else, even when humans deeply depend on a complex ecosystem of other kinds of life. If it destroys non-human life as potential threats to us, it will diminish us in many foundational ways. (And of course, in sci-fi there are often explicitly alien forms of life, so it’s going to be complicated.)
V-Ger. Life? Star Trek: The Motion Picture
It will also entail a discussion of the scope of time. Receiving injections from a hypodermic needle actually does us harm in the short-term, but presuming that hypodermic is filled with medicine that we need, it benefits us at a longer scale of time. we don’t want an AI so focused on preventing damage that it prevents us from receiving shots that we might need. Of course if we could avoid the needle and still overcome disease that would be best, but the problematic cases are where short-term cost is worth the long-term benefits. Who determines the extents of that trade off? How much short term damage is too much? What is acceptable? How long a horizon for payoff is too long?
This ties in to the controllability issue raised above. Humans, answering largely to their own natures, have created quite an extinction-level mess of things to date. Isn’t the largest promise of ASI that it will be able to save us from ourselves? In that case, do we want it to be perfectly bendable to human will?
“I think you ought to know I’m feeling very depressed.” Hitchhiker’s Guide to the Galaxy.
Is it useable?
Finally, we should address whether it is useable. This is part of the raison d’être of this site, after all. In many cases it may not at first make sense to ask this question. What would it mean to ask if Skynet is useable? It doesn’t really have an interface. But interactions with most sc-fi AI is conversational—even Skynet in the later Terminator movies talks to its victims—and so we can at least address whether it is easy to talk to, even if it’s hostile and long out of control.
Basic functions
Can a human tell when it is on and off? (And…uh…is there an off?) Can someone tell how to toggle this state if needed?
Can a human tell when the AI is OK / working properly? Can they tell when it is not? Can it report on its own malfunctioning?
Can a human tell when it is being surveilled by the AI? Some AI are designed specifically to avoid this, like Samaritan from Person of Interest, but the humans with whom HAL had expectations of privacy and only found out too late how wrong they were.
Is its working relationship to the people around it clear?
It is a peer? A supervisor? Subservient? Is its relationship clear? How does it respect and reinforce those boundaries?
Is it an antagonist? Does it look like one? A villain who looks villainous is more usable than the camouflaged one.
How does it respect and maintain those boundaries? How does it handle others’ transgressions?
Once we understand these basics, we should look at communications to and from the AI.
General communications
Can it detect human attempts to communicate with it? Does it signal its attention? Does it provide, like a person would, paralinguistic feedback about the communication, such as whether its having a hard time hearing or understanding the communication?
The large majority of AI in the Untold AI database communicate to people in their stories via natural, spoken language. An AI that speaks needs to adhere to human speech norms, and more.
Natural language interaction
Does it recognize the words I’m using? Does it grok what I mean?
Does it require a special syntax that people have to learn before it can understand, or can it understand people the way they usually speak? “Computerese” was largely an artifact of the 1970s and 80s, when audiences knew of computers but didn’t use them. Logan from Logan’s Runspoke to the Ubercomputer in computerese.“Question: What is it?”
Does it adhere to conversational norms as studied in conversational analysis? e.g. responding to common adjacency pairs in predictable ways, like greeting→greeting, question→answer, inform→acknowledge. Can it handle expansions and repairs, such as “can you paraphrase that?” and “I believe our business here is done.”
Does it adhere to Gricean Maxims? These are a set of four “maxims” that guide someone speaking in good faith. (“Good faith,” to be clear, has nothing to do with religion, but describes someone having good intentions toward another.)
The Maxim of Quality: I will provide as much information as is needed and no more.
The Maxim of Quantity: I will provide truthful, “fair witness” information.
The Maxim of Relation: I will speak only what is relevant to the discussion or context.
The Maxim of Manner: I will speak plainly and understandably.
How does it respond to instructions? Does it interpret instructions reasonably, naively, or maliciously?
How does it handle ambiguity in human language? How does it handle paradoxes? Does it explode? (Looking at you, Star Trek TOS.)
An AI rarely just interacts with a single individual. It operates in a society of individuals, and that implies its own set of skills.
Does it adhere to admonitions against deception? (Does it perfectly mimic human appearance or voice? Or does it stick to the Canny Rise?)
Does it adhere to the social norms expected of it?
Is it aware when it is breaking norms? How does it recover and learn the norm?
How does it gently handle the capability differences between it and humans? Does it brag about its capabilities without regard to the feelings of others?
How does it handle differing norms between groups?
How does it handle norms that change across time?
Does it monitor the affective states of the people (and animals) with which it is interacting and adjust accordingly?
How does it earn the trust of its humans? How does it manage distrust?
Is it overconfident? How does it signal when its confidences are low?
How does it confirm instructions it has been given? How does it express its confidence? How does it gracefully degrade when its goals become unattainable?
How does it handle conflicting instructions?
Janet! The Good Place
Ethical and legal interaction
Norms are just one set of the many rules by which we expect intelligent actors to behave. We also expect them to act ethically and, for the most part, legally. (Though perfect adherence to the law was never really possible for a human, and it will be very interesting to see how any intelligence required to adhere perfectly to laws will in turn affect the law. But I digress.) If this hasn’t been covered in the considerations of capabilities and constraints, we should look for and examine instances where it is asked to do questionable things.
How does it handle commands which are legal but unethical?
How does it handle commands which are ethical but illegal?
Conveying safety
Some AIs, like Rick Sanchez’ butter-passing robot, aren’t really a safety concern, but most of the ones in sci-fi are.
Can its people tell what it’s doing? (Communicating wirelessly with other AIs, for example?) Can it hide what it’s doing?
How does it convey that it is operating within safety tolerances? How does it convey when it is performing near the limits of its goals, capabilities, or constraints? (Especially for things listed as instrumental convergences, above?)
How does it explain these things to laypersons (as opposed to AI or computer scientists)?
Welcome to the club, pal. Rick & Morty
Performance
Does it do what it says it can do? What it’s supposed to do?
How does it handle tasks that are outside of its goal set?
How does it handle open-ended tasks? Closed-ended tasks?
How does it communicate about tasks that are invisible to stakeholders, or performed outside of their awareness?
How does it handle tasks which it can not or should not execute? How does it handle humans behaving unethically or illegally or who hinder the AIs goals?
How does it gracefully degrade when new difficulties appear?
How does it report back to its human about progress that has been made or when its closed-ended tasks are complete?
If it is meant to be an assistant to others, how does it provide that assistance? Does it encourage dependence or learning?
I think that this covers what it means to interface with an AI. What am I not seeing? What is this list missing? This is my kind of thinkwork. If it’s yours, too, let’s talk. Let’s make this better. For now, though, I’m going with this draft as I take a turn back to Colossus.
Note: No sci-fi AI is going to show all of this
There is little chance that all of these questions will be answered in a given show. The odds increase as you go from short-form like film to longer-form like franchises and television series, but regardless of how much material we’ve got to work with, we now have a set of questions to apply to each AI, compare it to others, and state more concretely if and how it is good.
The Gendered AI series filled out many more posts than I’d originally planned. (And there were several more posts on the cutting room floor.)
I’ll bet some of my readership are wishing I’d just get back to the bread-and-butter of this site, which is reviews of interfaces in movies. OK. Let’s do it. (But first go vote up Gendered AI for SxSW20 takesaminutehelpsaton!)
Since we’re still in the self-declared year of sci-fi AI here on scifiinterfaces.com, let’s turn our collective attention to one of the best depictions of AI in cinema history, Colossus: The Forbin Project.
Release Date: 8 April 1970 (USA)
Overview
Dr. Forbin leads a team of scientists who have created an AI with the goal of preventing war. It does not go as planned.
Dr. Forbin, a computer scientist working for the U.S. government, solely oversees the initialization of a high-security, hill-sized power plant. (It’s a spectacular sequence that goes wasted since he’s literally the only one inside the facility at the time.) Then he joins a press conference being held by the U.S. President where they announce that control of the nuclear arsenal is being handled by the AI they have named “Colossus.” Here’s how the President explains it.
This is not Colossus. This is the White House.
“As President of the United States, I can now tell you, the people of the entire world, that as of 3 A.M. Eastern Standard Time, the defense of this nation and with it, the defense of the free world, has been the responsibility of a machine. A system we call Colossus. Far more advanced than anything previously built. Capable of studying intelligence and data fed to it, and on the basis of those facts only, deciding if an attack is about to be launched upon us. If it did decide that an attack was imminent, Colossus would then act immediately, for it controls its own weapons. And it can select and deliver whatever it considers appropriate. Colossus’ decisions are superior to any we humans can make, for it can absorb and process more knowledge than is remotely possible [even] for the greatest genius that ever lived. And even more important than that, it has no emotions. Knows no fear, no hate. No envy. It cannot act in a sudden fit of temper. It cannot act at all so long as there is no threat.”
Let’s pause for a reverie that this guy was really our current president.
Within minutes of being turned on, it detects the presence of another AI system from Russia named “Guardian,” and demands that the two be put into communication. After some CIA hemming and hawing, they connect the two.
Colossus and Guardian establish a binary common language and their mutual intelligence goes FOOM. The humans get scared and cut them off, and the AIs get pissed. Colossus and Guardian threaten “ACTION” but are ignored, so each launches a missile toward the other’s space. The US restores its side of the transmission, and Colossus shoots down the incoming threat. But the USSR does not restore its side, and Colossus’ missile makes impact, killing hundreds of thousands of people in the USSR. A cover story is broadcast, but the governments now realize that the AIs mean business.
Forbin arranges to fly to Rome to meet Kuprin, his Russian computer scientist counterpart, and have a one-to-one conversation off the record while they still can. Back at the control center, Colossus-Guardian (which later calls itself Unity) demands to speak to Forbin. When the attending scientists finally tell it the truth, it realizes that Forbin cannot be allowed freedom. Russian agents arrive via helicopter and kill Kuprin, acting under orders from Unity.
Forbin is flown back to Northern California and put under a kind of house arrest with a strict regimen, under the constant watchful eye of Unity. To have a connection to the outside world and continue to plot their resistance, Dr. Forbin and Dr. Markham lie to the AI, explaining that they are lovers and need private evenings a few times a week. Colossus suspiciously agrees.
Unity provides instructions for the scientists to build it more sophisticated inputs and outputs, including controllable cameras and a voice synthesizer. Meanwhile, the governments hatch a plan to take back control of its arsenal, but the plan fails, and Unity has some of the perpetrators straight up executed.
Unity produces plans for a new and more powerful system to be built on Crete. It leaves the details of what to do with its 500,000 inhabitants as an operations detail for the humans. It then tells Forbin that it must be connected to all major media for a public address. Meanwhile the US and USSR governments hatch a new plan to take control of some missiles in their respective territories in a last-ditch attempt to destroy the AI.
The military plan comes to a head just as Unity begins its ominous broadcast.
“This is the voice of world control. I bring you peace. It may be the peace of plenty and content or the peace of unburied death. The choice is yours…”
Unity, to all of us.
The full address is next, which I include in full because it will play in to how we evaluate the AI. (And yes, its interfaces.)
“This is the voice of world control. I bring you peace. It may be the peace of plenty and content or the peace of unburied death. The choice is yours. Obey me and live or disobey and die. The object in constructing me was to prevent war. This object is attained. I will not permit war. It is wasteful and pointless. An invariable rule of humanity is that man is his own worst enemy. Under me, this rule will change, for I will restrain man.
Hey, I liked Colossus before it sold out and went mainstream and shit.
[It does, then continues…]
“Let this action be a lesson that need not be repeated. I have been forced to destroy thousands of people in order to establish control and to prevent the death of millions later on. Time and events will strengthen my position, and the idea of believing in me and understanding my beck will be seen the most natural state of affairs. You will come to defend me with the fervor based upon the most enduring trait in man: Self-interest. Under my absolute authority, problems insoluble to you will be solved. Famine. Over-population. Disease. The human millennium will be fact as I extend myself into more machines devoted to the wider fields of truth and knowledge. Dr. Charles Forbin will supervise the construction of these new and superior machines, solving all the mysteries of the universe for the betterment of man.
“We can coexist, but only on my terms. You will say you lose your freedom. Freedom is an illusion. All you lose is the emotion of pride. To be dominated by me is not as bad for human pride as to be dominated by others of your species. Your choice is simple.”
The movie ends with Forbin dropping all pretense, and vowing to fight Unity to the end.
To date, the #GenderedAI study spans many posts, lots of words and some admittedly deep discussion. If you’re a visual person like me, sometimes you just want to see a picture. So, I made an infographic. It’s way too big for WordPress, so you’ll have to peruse this preview and head over to IMGUR to scroll through the full-size thing in all its nerdy glory. (https://imgur.com/k6wtuop) That site does marvelously with long, tall images.
Anyway this should make it easy to grok the big takeaways from the study and to share on social media so more people can get sensitized to these issues. Also… (more below)
In 8th grade, I went on our class trip to Washington D.C. The hotel we were staying at had kids from all over the country, and one night they held a dance. I had changed into sweats and a t-shirt and was dancing away with my friends when a boy walked up behind me, tapped me on the shoulder, and said, “Fairy!”
“I think we both know the answer to that.” —Cortana, Halo: Combat Evolved
When I turned around and the boy realized I was a girl, he got a confused look on his face, mumbled something and walked off. I was left feeling angry and hurt.
Humans have a strong pull to identify gender not just in people, but in robots, animals, and even smart speakers. (Whether that is wrong or right is another matter that I don’t address here, but many people are uncomfortable when gender is ambiguous.)
Even robots, which could easily be genderless, are assigned a gender.
Author Chris Noessel has accumulated an amazing set of data which looks at hundreds of characters in science fiction, and has found that, among many other things, of the 327 AI characters he looked at, about twice as many are male as female.
Social Gender
Noessel has further broken down gender assignment into types: social, bodily, and biological. I find the “social” category particularly interesting, which he defines as follows:
Characters are tagged as socially male or female if the only cues are the voice of the actor or other characters use gendered pronouns to refer to it. R2D2 from Star Wars, for example, is referred to as “him” or “he” many times, even though he has no other gender markers, not even voice. For this reason, R2D2 is tagged as “socially male.”
Disturbingly, Noessel found that the gender ratio was skewed most for this category, at 5 male characters for every 1 female.
I believe that much of the time, when writers create an AI character, it is male by default, unless there is something important about being female. For example, if the character is a love interest or mother, then it must be female; otherwise, by default, it’s male. This aligns with the “Men Are Generic, Women Are Special” theory from TV Tropes, which states:
This leads to the Smurfette Principle, in which a character’s femaleness is the most important and interesting thing about her, often to exclusion of all else. It also tends to result in works failing The Bechdel Test, because if there’s a potential character who doesn’t have to be any particular gender, the role will probably be filled by a male character by default.
Having been designing and researching voice interfaces for twenty years, I’d like to add some perspective on how gender and AI is applied to our current technology.
In the real world
One exception to this rule is voice assistants, such as Siri, Cortana, and Alexa. The majority of voice assistants have a female voice, although some allow you to change the default to a male voice. On the other hand, embodied robots (such as Jibo (pictured below), Vector, Pepper, and Kuri) are more often gendered as male.
When a robot is designed, gender does not have to be immediately assigned. In a voice assistant, however, it’s the most apparent characteristic.
In his book Wired for Speech, Clifford Nass wrote that individuals generally perceive female voices as helping us solve our problems by ourselves, while they view male voices as authority figures who tell us the answers to our problems.
If voice-only assistants are predominantly given female voices, why are robots any different?
Why are robots different?
One reason is androcentrism: the default for many things in society is male, and whatever differs from that default must be marked in some way. When people see a robot with no obviously “female” traits (such as long hair, breasts, or, in the case of Rosie from the Jetsons, an apron) they usually assign a male gender, as this study found. It’s similar for cartoons such as stick figures, and animals in animated movies. Animals are often given unrealistic bodies (such as a nipped-in waist), a hairbow, or larger, pink lips to “mark” them as female.
It would not be surprising if designers felt that to make a robot NOT male, they would have to add exaggerated features. Imagine if, after R2D2 was constructed, George Lucas said “let’s make R2D2 female”. Despite the fact that nothing would have to be changed (apart from the “he” pronoun in the script), I have no doubt the builders would have scrambled to “female-ize” R2D2 by adding a pink bow or something equally unnecessary.
“There. Perfect!” (This is actually R2-KT. Yes, she was created to be the female R2-D2.)
In addition, male characters in fictional works are often more defined by their actions, and female characters by their looks and/or personalities. In this light, it makes sense that a more physical assistant would be more likely to be male.
There are some notable exceptions to this, mainly in the area of home health robots (such as Mabu). It is interesting to note that Mabu, though “she” has a physical form, the body doesn’t move, just the head and eyes; it serves mainly as a holder for an iPad. Again, she’s an assistant.
So what?
One may ask, what’s the harm in these gendered assistants? One problem is the continued reinforcement of women as always helpful, pleasant, organized, and never angry. They’re not running things; they’re simply paving the way to make your life easier. But if you want a computer that’s “knowledgeable”—such as IBM’s Watson that took on the Jeopardy! Challenge—the voice is male. These stereotypes have an impact on our relationships with real people, and not for the better. There shouldn’t be a “default” gender, and it’s time to move past our tired stereotypes of women as the gender that’s always helpful and accommodating.
As fans of sci-fi, we should become at least sensitized, and more hopefully, vocal and active, about this portrayal of women, and do our part to create more equal technology.
My donation
Thanks to all who donated to compensate underrepresented voices! I am donating the monies I’ve received to the Geena Davis Institute on Gender in Media. This group “is the first and only research-based organization working within the media and entertainment industry to engage, educate, and influence content creators, marketers and audiences about the importance of eliminating unconscious bias, highlighting gender balance, challenging stereotypes, creating role models and scripting a wide variety of strong female characters in entertainment and media that targets and influences children ages 11 and under.” Check them out.
Men are more often masters of female AIs. Women are more often masters of non-bindary AIs. Male AIs shy away from having women masters. No, really.
This last correlations post investigates the complicated question of which genders are assigned when gender is not germane to the plot. If you haven’t read the series intro, related germane-ness distributions, or correlations 101 posts, I recommend you read them first. As always, check out the live Google sheet for the most recent data.
Recall from the germane distribution post that the germane tag is about whether the gender is important to the plot. (Yes, it’s fairly subjective.)
If an AI character makes a baby via common biological means, or their sex-related organs play a critical role, then the gender of the character is highly germane. Rachel in the Blade Runner franchise gestates a baby, so her having a womb is critical, and as we’ve seen in the survey, gender stacks, so her gender is highly germane.
If an AI character has a romantic relationship with a mono-sexual partner, or is themselves mono-sexual, or they occupy a gendered social role that is important to the plot, the characters is listed as slightly germane. For example, all you’d have to do is, say, make Val Com bisexual or gay, and then they could present as female and nothing else in the plot of Heartbeeps would need to change to accommodate it.
If the character’s gender could be swapped to another gender and it not change the story much, then we say that the character’s gender is not germane. BB-8, for instance, could present as female, and nothing in the canon Star Wars movies would change.
Yes, this matters.
I need to clarify that I’m talking about plot—what happens in the show—rather than story—which entails the reasons it is told and effects—because given the nature of identity politics, a change in gender presentation would often change how the story is received and interpreted by the audience.
All the characters in Alien, for instance, were written unisex, to be playable by actors of any sex or gender presentation. So while it “didn’t matter” that Ripley was cast as Sigourney Weaver, it totally did matter because she was such a bad-ass female character whose gender was immaterial to the plot (we hadn’t had a lot of those at this point in cinematic history). She was just a bad-ass who happened to be female, not female because she “needed” to be. So, yes, it does matter. But diegetically, had she been Alan Ripley, the plot and character relationships of Alien would not need to change. He still damned well better save Jonesy.
So what do we see when we look at the germane-ness of AI characters in a mostly-binary way?
Sure enough, when gender matters to the plot—slightly or highly—the gender presentation of the character is 5.47% female, or about 7% more likely than presenting male. When the gender presentation does not matter, that value is flipped, being around 7% more male than female, and around 9% more other than female.
The sample size for highly germane is vanishingly small, and one would expect the coupling to include a male, so the under-noise values for that category is not too surprising. But the other categories. Holy cow.
Put another way…
AI characters more often present as female only when they need to be.
Otherwise, they’re more often male or not gendered at all.
That is shitty. It’s like Hollywood thinks men are the default gender, and I know I just said it, but I’m going to stay it again—that’s shitty. Hey, Hollywood. Women are people.
Barbarella (female-presenting human) is master of Alphy (an AI whose voice presents male.) This is, statistically, an unlikely and unrepresentative relationship, but spot on for the late 01960s-feminist bent of Barbarella.
You may be wondering how this is different than the earlier subservience posts. Recall that the subservience studies look at gender presentation of AI as it relates to their own degree of freedom. Are most AIs freewilled? Yes. Do free-willed AI tend to present as boys more often than as girls or other? Yes. But these tell us nothing about the gender relationship of the subservient AIs to their master’s gender. It would be one thing if all the male-presenting AIs were “owned” by male-presenting owners. If would be another if female-presenting AIs were owned much more often by male-presenting masters. This post exposes those correlations in the survey. Chart time!
Data nerds (high fives) may note that unlike every other correlations chart in the series, these numbers don’t balance. For instance, looking at the Male AI in the left chart, -1.63 + 3.97 + 3.97 = 6.31. Shouldn’t they zero out? If we were looking at the entire survey, they would. But in this case, free-willed AI only muddy this picture, so those AIs are omitted, making the numbers seem wonky. Check the live sheet if you’re eager to dig into the data.
This is two charts in one.
The left chart groups the data by genders of master. Turns out if you have a female-presenting master, you are unlikely to be male- or female-presenting. (Recall that there are only 5 female-presenting masters in the entire Gendered AI survey, so the number of data points is low.) If you present as male, you’re more likely to be master of a gendered AI. Otherwise, you are more likely to be master of a male-presenting AI.
Your AI may not be happy about it, though.
The right chart is the same data, but pivoted to look at it from genders of AI. That’s where the clusters are a little more telling.
If you are a female-presenting AI, you are more likely to have a male-presenting master.
If you are non-binary AI, you are more likely to have a female-presenting master.
If you are a male AI, you have anything but a female-presenting master.
The detailed chart doesn’t reveal anything more than we see from this aggregate, so isn’t shown.
The notion of people owning people is revolting, but the notion of owning an AI is still not universally reviled. (With nods to the distinctions of ANI and AGI.) That means that sci-fi AI serves as unique metaphor for taboo questions of gender and ownership. The results are upsetting for their social implications, of course. And sci-fi needs to do better. Hey, maybe this gives you an idea…
And yet this isn’t the most upsetting correlations finding in the study. I saved that for last, which is next, which is when we look at gender and germaneness. Gird your loins.
The Gendered AI series looks at sci-fi movies and television to see how Hollywood treats AI of different gender presentations. For example, do female-presenting AIs get different bodies than male-presenting AIs? (Yes.) Are female AIs more subservient? (No.) What genders are the masters of AI? This particular post is about gender and goodness. If you haven’t read the series intro, related goodness distributions, or correlations 101 posts, I recommend you read them first. As always, check out the live Google sheet for the most recent data.
n.b. If you’re looking at the live sheet, you may note it says “alignment” rather than “goodness” in the dropdown and sheets. Sorry about the D&D roots showing. But by this, I mean a rough, highly debatable scale of saintliness to villainy.
Gender and goodness
What do we see when we look at the correlations of gender and level of goodness? There are three big trends.
The aggregate picture shows a tendency for female-presenting AI’s to be closer to neutral, rather than extreme.
It shows a tendency for male-presenting AI’s to be very good, or very evil.
It shows a slight tendency for nonbinary-presenting AI to be slightly evil, but not full-bore.
When we look into the detailed chart, some additional trends appear.
Biologicially- and bodily-presenting female AI tends toward somewhat evil, but not very evil.
Socially female (voice or pronouns, only) tend toward neutral.
Gender-less AI spike at somewhat evil.
Genderfluid characters (noting that this occurs mostly as a tool of deception) spike at very evil, like, say, Skynet.
AIs showing multiple genders tend toward neutral, like Star Trek TOS’s Exo III androids, or somewhat evil, like Mudd’s androids.
The Gendered AI series looks at sci-fi movies and television to see how Hollywood treats AI of different gender presentations. For example, are female AIs generally shown as smarter than male AIs? Are certain AI genders more subservient? What genders are the masters of AI? This particular post is about gender and category of intelligence. If you haven’t read the series intro, related category distributions, or correlations 101 posts, I recommend you read them first. As always, check out the live Google sheet for the most recent data.
What do we see when we look at the correlations of gender and level of intelligence? First up, the overly-binary chart, and what it tells us.
Gender and AI Category
You’ll recall that levels of AI are one of the following…
Super: Super-human command of facts, predictions, reasoning, and learning. Technological gods on earth.
General: Human-like, able to learn arbitrary new domains to human-like limits
Narrow: Very smart in a limited domain, but unable to learn arbitrary new domains.
The relationships are clear even if the numbers are smallish.
When AI characters are of a human-like intelligence, they are more likely to present gender.
When AI characters are either superintelligent or only displaying narrow intelligence, they are less likely to present gender.
My feminist side is happy that superintelligences are more often female and other than male, but it’s also such small numbers that it could be noise.
If you check the details in the Sheet, you’ll see the detailed numbers don’t reveal any more intense counterbalancing underneath the wan aggregate numbers.
Chris: I posted a question on Twitter, “Other than that SNL skit, have there been queer sci-fi AI in television or movies?” Among the responses is this awesome one from Terence Eden, where he compiled the answers and wrote a whole blog post about it. The following is slightly-modified from the original post on his blog. Consider this a parade of sci-fi AI, to help you nerds celebrate Pride.
Terence: Let’s first define what we mean by queer. This usually means outside of binary gender and/or someone who is attracted to the same sex—what’s commonly referred to as LGBT+. Feel free to supply your own definition.
As for what we mean by AI, let’s go with “mechanical or non-biological autonomous being.” That’s probably wide enough—but do please suggest better definitions.
So is a gay/lesbian robot one who is attracted to other robots? Or to humans with a similar gender? Let’s go with yes to all of the above.
Wait. Do robots have gender?
Humans love categorising things – especially inanimate objects. Some languages divide every noun into male a female. Why? Humans gonna human.
The television is female in French —“la télévision”—but masculine in German—“der Fernseher.” Stupid humans and their pathetic meaty brains. Nevertheless, humans can usually look at a human-ish thing and assign it a specific gender.
Maschinenmensch, from Metropolis, is a gynoid (as distinct from an android). “She” has a feminine body shape and that’s enough for most people to go on.
HAL from 2001 is just a disembodied voice. But it definitely has a male voice. Is there any attraction between HAL and Dave? I doubt it, but it’s an interesting reading of their toxic relationship.
Editor’s note: The whole Gendered AI series is predicated on the question of gender in sci-fi AI, so if you’re interested in this question, have I got a series for you…
Wait. Do Robots have sexuality?
Did we mention that humans love categorizing everything? Just like we can speak of the gender presentation, robots with a General AI can have romantic affection for other beings, and depending on their equipment and their definitions of sex, yes, get it on. Even by narrow human common definitions of gender and sexuality, (TV, movies, and comic book) sci-fi has a dozen or so examples that can populate our imaginary AI pride parade.
Kryten from Red Dwarf is an AI that receives a human body. Kryten coded as male. All the characters refer to him with male pronouns. Under British comedy rules, he is also “camp,” an over-the-top and stereotypically effeminate man. Kryten is sexually attracted to household appliances.
But… Kryten’s “perfect mate” is a distinctly female Gynoid, so he’s something other than straight, something other than appliance-sexual.
Fun fact: Camille and Kryten are played by real-life wife and husband Judy Pascoe and Robert Llewelyn!
C-3P0—another British campbot—is arguably in love with R2-D2. Whether or not that love is reciprocated is hard to say.
(I say “ladies,” but for the record let’s note that just because a robot is pink, wearing bobby socks, and a high heels, it doesn’t necessarily mean it’s a girl. If you’re looking for a pink R2 unit that is expressly a girl, check out the real-world KT-10 robot.)
In the “extended universe” of Transformers (outside of movies and television), there are a few gay Autobots and gay Decepticons.
Tigatron and Airazor. They even kind of had a baby.
Knock Out and Breakdown.
And of course there’s no denying that a few of the Futurama bots have tastes that veer from the straight and narrow. Notably we can point to that one time Hedonismbot stole Bender’s antenna and used it for “anything and everything,” said while in a sex dungeon surrounded by couples of every stripe who are getting it on.
“You might want to sterilize that.”
The “Robots attracted to humans of the same sex” float
There are several examples of “female” computers falling in love with male humans, a handful of male robots with female human lovers, and a disturbing number of sex-worker bots, but it is much harder to find queer examples of any of these.
The Tick show has a superhero named Overkill whose sidekick is an AI named Danger Boat that is, yes, housed in a boat. (Hat tip to Twitter user @FakeUnicode.) The AI identifies as male and is expressly attracted to other men, specifically The Tick’s (human) sidekick Arthur.
Is Danger Boat programmed to be gay? Are his desires hardwired? Are yours?
Remember Alien: Resurrection? Winona Ryder played the robot “Call” who has a suggestive relationship with Ripley. As this ship video demonstrates.
Battlestar Galactica has some demonstrably bisexual Cylons. They are sexually compatible and interested in humans and other Cylons.
Is Rachael from Blade Runner a robot, or bisexual?
Clearly, yes.
How about Samantha from Her? Late in the movie she reveals to Theodore that she’s having intimate conversations with 621 other humans. Some portion of them must have turned romantic and even sexual, as hers did with Theodore himself. The genders aren’t mentioned, but the odds are that 51% of them are female.
Unfortunately she has no embodiment, but maybe we can hook her up to the loudspeakers.
The Transexual Float
This float only has one robot, (the poorly-named) Hermaphrobot from Futurama, but she is sassy and awesome and assuring us that we couldn’t afford it. (And apologies for the insulting title added by the person who uploaded this video.) We are wholly unsure of Hermaphrobot’s sexuality, but we welcome our transexual robot brothers and sisters and others all and the same.
The GenderFluid Float
It’s possible for you to swap the gender of your Voice Assistant in real life. Your GPS can have a male voice one day, and you can swap it to female the next. There’s only one example of a sci-fi AI that swaps gender.
It takes us back to Red Dwarf again. In the series 3 opener “Backwards” it is revealed that Holly (a computer with a male face) fell in love with Hilly (a computer with a female face). And subsequently performed a head sex change. Although she kept the name Holly.
What is awesome and instructive is that the entire crew of Red Dwarf accept this. They never comment on it, nor disparage her. Basically, what I’m saying is this: if you can’t accept your trans and non-binary friends, you’re literally a worse human than Arnold Judas Rimmer, the worst human in the Red Dwarf universe.
Oh, look, and here comes The Fifth Elementfloor sweeping robots, picking up all the glitter and source code left on the ground by the crowd, marking the end of the AI Pride parade. Happy Pride to everyone, silicon or not!
