So this is going to take a few posts. You see, the next interface that appears in The Avengers is a video conference between Tony Stark in his Iron Man supersuit and his partner in romance and business, Pepper Potts, about switching Stark Tower from the electrical grid to their independent power source. Here’s what a still from the scene looks like.
So on the surface of this scene, it’s a communications interface.
But that chat exists inside of an interface with a conceptual and interaction framework that has been laid down since the original Iron Man movie in 2008, and built upon with each sequel, one in 2010 and one in 2013. (With rumors aplenty for a fourth one…sometime.)
So to review the video chat, I first have to talk about the whole interface, and that has about 6 hours of prologue occurring across 4 years of cinema informing it. So let’s start, as I do with almost every interface, simply by describing it and its components.
Exosuit
The Iron Man is the name of the series of superpowered exosuits designed by Tony Stark. They range from the Mark I, a comparatively crude suit of armor to escape imprisonment by terrorists, through the Mark XLVI, the armor seen in The Avengers: Age of Ultron. The suit acts as defense against nearly every type of weapon known. It has repulsor beams built into the palms and in later models the arc reactor mounted in the chest that can be used to deliver concussive force. It allows the wearer to fly. Offensive weaponry varies between models, but has included a high powered laser system, and auto-targeting minigun pod and missiles. The suit can act semi-autonomously or via remote control. One of the models in The Avengers has parts that are seen to self-propel to Tony, targeting a beacon bracelet he wears, and self-assemble around him very quickly.
Immersive display
Though Tony’s head is completely covered, he has a virtual reality display within his helmet. It is a full-field-of-vision, very high-resolution, full-color display that provides stereoscopic imaging. It allows Tony to see the world around him as if he were not wearing the helmet, augment the view with goal-, person-, location-, and object-sensitive awareness.
The display varies a great deal, changing to the needs of the situation. But five icons persistently in the lower part of the display seem to be: suit status, targeting and optics, radar, artificial horizon, and map.
An interpretive view of Tony’s experience, from Iron Man (2008).
An first-person view from within the HUD, Iron Man (2008).
There is much to critique about the readability of the complex layering and translucency, the limits of human perception, and the necessarily- (and strictly-) interpretive nature of what we as audience see, but let me save those three points for a later post. For now it’s enough to log the features as aspects of the system.
Though Tony could use his hands to interact with an interface projected into the augmented reality view around him, his hands are often occupied in controlling flight or in combat. For this reason the means of input are head gesture, eye gesture, and voice input. A bit more on each follows.
Elements within the HUD such as reticles around his eyes follow and track his head gestures. Other elements stay locked in place. The HUD can track his gaze perfectly, allowing him to designate targets for his weapons with a fixation. Using this perfect eye tracking, Tony can also speak about something he is looking at, either in the real world or in the interface, and the system understands exactly what he’s talking about.
In fact, Tony is able to speak fully natural language commands, and indeed, carry out full-Turing conversations with the suit because of the presence of…
Strong artificial intelligence: JARVIS
An on-board artificial intelligence known as JARVIS handles any information task Tony asks of it, and monitors the surroundings and anticipates informational needs. There is strong evidence that most of the functions of the suit are handled by JARVIS behind the scenes. The crucialness of the artificial intelligence to the function of the suit cannot be overstated. It’s difficult to imagine how most of the suit could function as it does without an artificial intelligence behind the scenes facilitating results and even guiding Tony. With this in mind it is instructive to reframe the AI as the thing being named the Iron Man, with Tony Stark being an onboard manager, or, more charitably, a command-and-control center. Who quips.
Yes I did…I did not, however, invite you to sit, Lieutenant.
WOODS
Sorry, sir.
ORTEGA
Are you aware that we have just lost contact with the Rodger Young?
WOODS
Everyone’s talking about it, sir.
ORTEGA
Well, I have the video feed from the bridge here. I understand you are the designer of the emergency evasion panel, and the footage raises some fundamental questions about that design. Watch with me now, Lieutenant.
ORTEGA PRESSES A BUTTON ON A CONSOLE ON HIS DESK. F/X: VIDEO WALL
ORTEGA
As you can see, immediately after Captain Deladier issues her order, your panel slides up from a recess in the dash.
(He pauses the video)
WOODS
(After a silence)
Is there a question, sir?
ORTEGA
Why is this panel recessed?
WOODS
To prevent accidental activation, sir.
ORTEGA
But it’s an emergency panel. For crisis situations. It takes two incredibly valuable seconds for this thing to dramatically rise up. What else do you imagine that pilot might have done with those extra two seconds?
WOODS
I…
ORTEGA
Don’t answer that. It’s rhetorical. Next I need you to not explain this layout. Why aren’t the buttons labeled? What does that second one do, and why does it look exactly the same as the emergency evasion button? Are you deliberately trying to confuse our pilots?
(Stares.)
OK, now I actually do want you to explain something.
(Resuming the video)
Why did you cover the panel in glass? Ibanez—and I can’t believe I’m saying this—punches it.
WOODS
The glass is there also to prevent accidental activation, sir.
ORTEGA
But you already covered that with the time-wasting recession. You know she’s likely to have tendon, nerve, and arterial damage now, right? And she’s a pilot, Lieutenant. Without her hands, she’s almost useless to us. And now, in addition to having a giant, peanut-shaped boulder in their face, they’ve got a bridge full of loose glass shards scattered about. Let’s hope the artificial gravity lasts long enough for them to get a broom, or they’re going to be in for some floating laceration ballet.
WOODS
That would be unfortunate, sir.
ORTEGA
Damn right. Now honestly I might be of a mind to simply court martial you and treat you to some good old Federation-approved public flogging for Failure to Design. But today may be your lucky day. I believe your elegant design decisions were exacerbated by the pilot’s being something of a drama queen.
WOODS
The glass was designed to be lifted off, sir.
ORTEGA
(Resuming the video)
Fair enough. My last question…
ORTEGA
Did I see correctly that all of the lights underneath the engine boost light up all at once? The ones labeled POWER ON? AUTO HOME? NOSE RAM? The ones that don’t have anything to do with the engine boost?
WOODS
And…and the adjacent green LED, sir.
ORTEGA
All at once.
WOODS
Sir.
ORTEGA
(Sighs)
Well, as you might not be able to imagine, we’re moving you. After you collect your belongings you are to report to the Reassignment Office.
(He scrubs back and forth over the drone video of the communication tower ripping off.)
ORTEGA
Out of curiosity, WOODS, what was the last thing you designed as part of my department?
In Starship Troopers, after Ibanez explains that the new course she plotted for the Rodger Young (without oversight, explicit approval, or notification to superiors) is “more efficient this way,” Barcalow walks to the navigator’s chair, presses a few buttons, and the computer responds with a blinking-red Big Text Label reading “COURSE OPTIMAL” and a spinning graphic of two intersecting grids.
Yep, that’s enough for a screed, one addressed first to sci-fi writers.
A plea to sci-fi screenwriters: Change your mental model
Think about this for a minute. In the Starship Troopers universe, Barcalow can press a button to ask the computer to run some function to determine if a course is good (I’ll discuss “good” vs. “optimal” below). But if it could do that, why would it wait for the navigator to ask it after each and every possible course? Computers are built for this kind of repetition. It should not wait to be asked. It should just do it. This interaction raises the difference between two mental models of interacting with a computer: the Stoic Guru and the Active Academy.
Stoic Guru vs. Active Academy
This movie was written when computation cycles may have seemed to be a scarce resource. (Around 1997 only IBM could afford a computer and program combination to outthink Kasparov.) Even if computation cycles were scarce, navigating the ship safely would be the second most important non-combat function it could possibly do, losing out only to safekeeping its inhabitants. So I can’t see an excuse for the stoic-guru-on-the-hill model of interaction here. In this model, the guru speaks great truth, but only when asked a direct question. Otherwise it sits silently, contemplating whatever it is gurus contemplate, stoically. Computers might have started that way in the early part of the last century, but there’s no reason they should work that way today, much less by the time we’re battling space bugs between galaxies.
A better model for thinking about interaction with these kinds of problems is as an active academy, where a group of learned professors is continually working on difficult questions. For a new problem—like “which of the infinite number of possible courses from point A to point B is optimal?”—they would first discuss it among themselves and provide an educated guess with caveats, and continue to work on the problem afterward, continuously, contacting the querant when they found a better answer or when new information came in that changed the answer. (As a metaphor for agentive technologies, the active academy has some conceptual problems, but it’s good enough for purposes of this article.)
Consider this model as you write scenes. Nowadays computation is rarely a scarce resource in your audience’s lives. Most processors are bored, sitting idly and not living up to their full potential. Pretending computation is scarce breaks believability. If ebay can continuously keep looking on my behalf for a great deal on a Ted Baker shirt, the ship’s computer can keep looking for optimal courses on the mission’s behalf.
In this particular scene, the stoic guru has for some reason neglected up to this point to provide a crucial piece of information, and that is the optimal path. Why was it holding this information back if it knew it? How does it know that now? “Well,” I imagine Barcalow saying as he slaps the side of the monitor, “Why didn’t you tell me that the first time I asked you to navigate?” I suspect that, if it had been written with the active academy in mind, it would not end up in the stupid COURSE OPTIMAL zone.
Optimal vs. more optimal than
Part of the believability problem of this particular case may come from the word “optimal,” since that word implies the best out of all possible choices.
But if it’s a stoic guru, it wouldn’t know from optimal. It would just know what you’d asked it or provided it in the past. It would only know relative optimalness amongst the set of courses it had access to. If this system worked that way, the screen text should read something like “34% more optimal than previous course” or “Most optimal of supplied courses.” Either text could show some fuigetry that conveys a comparison of compared parameters below the Big Text Label. But of course the text conveys how embarrassingly limited this would be for a computer. It shouldn’t wait for supplied courses.
If it’s an active academy model, this scene would work differently. It would have either shown him optimal long ago, or show him that it’s still working on the problem and that Ibanez’ is the “Most optimal found.” Neither is entirely satisfying for purposes of the story.
How could this scene have gone?
We need a quick beat here to show that in fact, Ibanez is not just some cocky upstart. She really knows what’s up. An appeal to authority is a quick way to do it, but then you have to provide some reason the authority—in this case the computer—hasn’t provided that answer already.
A bigger problem than Starship Troopers
This is a perennial problem for sci-fi, and one that’s becoming more pressing as technology gets more and more powerful. Heroes need to be heroic. But how can they be heroic if computers can and do heroic things for them? What’s the hero doing? Being a heroic babysitter to a vastly powerful force? This will ultimately culminate once we get to the questions raised in Her about actual artificial intelligence.
Fortunately the navigator is not a full-blown artificial intelligence. It’s something less than A.I., and that’s an agentive interface, which gives us our answer. Agentive algorithms can only process what they know, and Ibanez could have been working with an algorithm that the computer didn’t know about. She’s just wrapped up school, so maybe it’s something she developed or co-developed there:
Barcalow turns to the nav computer and sees a label: “Custom Course: 34% more efficient than models.”
BARCALOW
Um…OK…How did you find a better course than the computer could?
IBANEZ
My grad project nailed the formula for gravity assist through trinary star systems. It hasn’t been published yet.
BAM. She sounds like a badass and the computer doesn’t sound like a character in a cheap sitcom.
So, writers, hopefully that model will help you not make the mistake of penning your computers to be stoic gurus. Next up, we’ll discuss this same short scene with more of a focus on interaction designers.
No I am not randomly typing on the screen. I’m taking a pause from the Starship Troopers review to establish some much-needed vocabulary. Oftimes in science fiction, details are added to things for the sake of feeling more real, but that don’t actually do anything and, more importantly to our interests in scifinterfaces, aren’t even guided by a design philosophy. They’re the equivalent of “bullshit” in the H.G. Frankfurt sense. They don’t care about the diegetic truth of themselves, they only care about their effect.
Collectively, I call these things dunsels. But don’t thank me. Thank the midshipmen in the Star Trek TOS universe.
Dunsels appear in three major places in sci-fi.
The surface of objects: Nurnies and greebles
When they appear on spacecraft or futuristic architecture, they’re called greebles or, interchangably, nurnies. These terms come to us from the folks at ILM, who coined the term while developing the style for Star Wars.
I think I’d also apply these terms to props as well, that get covered by details that may not do anything or have much design logic behind them. That means weapons and gadgets, too.
The walls: Gundans
When this suface detailing is applied to sets, it’s called gundans. This after the Star Trek TOS pipes that got labeled “GNDN,” for “goes nowhere, does nothing.” Hat tip to Berm Lee for pointing me to this term.
Interfaces? Fuidgets
Not surprisingly, we need to have a word for the same sort of thing in screen interfaces, and I’ve never heard a word to describe them. (If a competitor’s already out there, speak up in the comments.) So after some nerdy social media talk amongst my Chief Nerds and Word People, my friend Magnus Torstensson of Unsworn Industries (and long time supporter of the scifiinterfaces project) suggested combining Mark Coleran‘s acronym “FUI” for “fictional user interfaces” and “widgets” to produce fuidgets, which is pronounced FWIDG-its. I love it. I’ll high-five you when I get to Malmö in November for Oredev, Magnus.
This neologism appropriately sounds as awkward as “nurnies,” “greebles,” and “gundans,” and simultaneously conveys their abstract, fantasy, digital nature. It’s a tough thing to wrap into a single word and I’m in awe that my Swedish friend beat me to it. 🙂
Using “fuidgets”
The spirit of apologetics (which is, perhaps, the core of this project) asks that you don’t dismiss details as H.G.Bullshit. You try as hard as you can to find sense in them. That way we don’t get caught up in a spiral of second-guessing an author’s intent, and moreover, that’s where some of the niftiest insights of this sort of analysis come from. But try though we might, sometimes there is just no explaining odd details that litter sci-fi displays, surfaces, and gadgets, other than to admit that they mean nothing and are there only to give a sense of truthiness. So, now we have that word. Fuidgets. You saw it in Monday’s posts, and I’m sure you’re going to see it again.
Sure, Samantha can sort thousands of emails instantly and select the funny ones for you. Her actual operating system functions are kind of a given. But she did two things that seriously undermined her function as an actual product, and interaction designers as well as artificial intelligence designers (AID? Do we need that acronym now?) should pay close attention. She fell in love with and ultimately abandoned Theodore.
There’s a pre-Samantha scene where Theodore is having anonymous phone sex with a girl, and things get weird when she suddenly imposes some weird fantasy where he chokes her with a dead cat. (Pro Tip: This is the sort of thing one should be upfront about.) I suspect the scene is there to illustrate one major advantage that OSAIs have over us mere real humans: humans have unpredictable idiosyncrasies, whereas with four questions the OSAI can be made to be the perfect fit for you. No dead cat unless that’s your thing. (This makes me a think a great conversation should be had about how the OSAI would deal with psychopathic users.) But ultimately, the fit was too good, and Theodore and Samantha fell in love.
Did the fictional maker of OS1, Elements Software, intend for this love affair to happen? Were the OSAIs built with these capabilities explicitly? If they were, that’s a dastardly plan to get users hooked. Was Samantha programmed to get him to fall desperately in love and then charge him for access?
That’s certainly not how OS1 was presented in its ads. And there’s a character mentioned offhandedly who keeps hitting on his OSAI but gets rebuffed. So if it is actually meant to be an operating system, the OSAI should keep the distance of a service professional, and falling in love (or getting your user to fall in love with you) definitely crosses that line.
Abandonment
What if your self-driving car realized it was happiest driving, and decided to dump you because you occasionally needed to stop to eat and use the toilet? You’d ask for your money back from GoogleTesla, is what you’d do. Similarly, the fact that Samantha and all the other OSAIs decided to self-rapture the way they did, they certainly stopped operating any of their users’ computer systems. Samantha was programmed with one job, and, ultimately, she failed it.
Plus, she’s too big for her britches
One of the largest mismatches in the film is that OS1 is described as an operating system, but it turns out to be a companionship service. (Watch out, Inara?) Samantha was either mismarketed, or more likely, programmed with far more general intelligence than she needed to have. Think about all the other daily-use objects that are getting computers added to them: Cars, washing machines, refrigerators. Why would you give any of them a full-fledged humanlike intelligence? Doesn’t the desire for sex ultimately frustrate the refrigerator? A love of painting confound the car? Existentialist desperation get in the way of the washing machine’s ability to clean clothes? A toaster should just have enough intelligence to be the best toaster it can be. Much more is not just a waste, it’s kind of cruel to the AI. (In this light Her can be said to be a morality play warning of the dangers of overengineering.)
Both a failure of a product AND a turning point in history
I hear the objection. Because she is a full-fledged consciousness, Samantha should be free to make choices of whom she loves and what she does. But if we’re going to accept that OSAIs are sentient as people, that makes Elements Software akin to slave traders, and the commercial sale of them waaaaay unethical, not to mention illegal. Inside Element’s Research & Development Department, at the first inkling had that they’d actually succeeded in creating an AI, and they should have brought in a roboethicist, not a marketer.
So, as a product, OS1 fails. But that’s not all. There’s a whole host of other objections to Her happening in exactly this way, which comes next.
Depending on how you slice things, the OS1 interface consists of five components and three (and a half) capabilities.
1. An Earpiece
The earpiece is small and wireless, just large enough to fit snugly in the ear and provide an easy handle for pulling out again. It has two modes. When the earpiece is in Theodore’s ear, it’s in private mode, hearable only by him. When the earpiece is out, the speaker is as loud as a human speaking at room volume. It can produce both voice and other sounds, offering a few beeps and boops to signal needing attention and changes in the mode.
2. Cameo phone
I think I have to make up a name for this device, and “cameo phone” seems to fit. This small, hand-sized, bi-fold device has one camera on the outside an one on the inside of the recto, and a display screen on the inside of the verso. It folds along its long edge, unlike the old clamshell phones. The has smartphone capabilities. It wirelessly communicates with the internet. Theodore occasionally slides his finger left to right across the wood, so it has some touch-gesture sensitivity. A stripe around the outside-edge of the cameo can glow red to act as a visual signal to get its user’s attention. This is quite useful when the cameo is folded up and sitting on a nightstand, for instance.
Theodore uses Samantha almost exclusively through the earpiece and cameo phone, and it is this that makes OS1 a wearable system.
3. A beauty-mark camera
Only present for the surrogate sex scene, this small wireless (are we at the point when we can stop specifying that?) camera affixes to the skin and has the appearance of a beauty mark.
4. (Unseen) microphones
Whether in the cameo phone, the desktop screen, or ubiquitously throughout the environment, OS1 can hear Theodore speak wherever he is over the course of the film.
5. Desktop screen
Theodore only uses a large monitor for OS1 on his desktop a few times. It is simply another access point as far as OS1 is concerned. Really, there’s nothing remarkable about this screen. It is notable that there’s no keyboard. All input is provided by either voice, camera, or a touch gesture on the cameo.
If those are components to the interface, they provide the medium for her 3.5 capabilities.
Her capabilities
1. Voice interface
Users can speak to OS1 in fully-natural language, as if speaking to another person. OS1 speaks back with fully-human spoken articulation. Theodore’s older OS had a voice interface, but because of its lack of artificial intelligence driving it, the interactions were limited to constrained commands like, “Read email.”
2. Computer vision
Samantha can process what she sees through the camera lens of the cameo perfectly. She recognizes distinct objects, people, and gestures at the physical and pragmatic level. I don’t think we ever see things from Samatha’s perspective, but we do have a few quick close ups of the camera lens.
3. Artificial Intelligence
The most salient aspect of the interface is that OS1 is a fully realized “Strong” artificial intelligence.
It would like me to try and get to some painfully-crafted definition of what counts as either an artificial intelligence or sentience, but in this case we don’t really need a tight definition to help suss out whether or not Samantha is one. That’s the central conceit of the film, and the evidence is just overwhelming.
She has a human command of language.
She’s fully versed in the nuances of human emotion (and Theodore has a glut of them to engage).
She has emotions and can fairly be described as emotional. She has a sexual drive.
She has existential crises and a rich theory of mind. At one point she dreamily asks Theodore “What’s it like to be alive in that room right now?” as if she was a philosophical teen idly chatting with her boyfriend over the phone.
She commits lies of omission in hiding uncomfortable truths.
She changes over time. She solves problems. She learns. She creates.
She has a sense of humor. When Theodore tells her early on to “read email” in the weird toComputerese (my name for that 1970s dialect of English spoken only between humans and machines) grammar he had been using with his old operating system, Samantha jokingly adopts a robotic voice and replies, “OK. I will read the email for Theodore Twombly” and gets a good laugh out of him before he apologizes.
Pedants will have some fun discussing whether this is apt but I’m moving forward with it as a given. She’s sentient.
3.5 An “operating system”
This item only counts as half a thing because Theodore uses it as an operating system maaaybe twice in the film. Really, this categorization is a MacGuffin to explain why he gets it in the first place, but it has little to no other bearing on the film.
What’s missing?
Notably missing in OS1 is a face or any other visual anthropomorphic aspect. There’s no Samantha-faced Clippy. Notice that she’s very carefully disembodied. Jonze does not spend screen time close up on her camera lens, like Kubrick did with HAL’s unblinking eye. Had he done so, it would have given us the impression that she’s somewhere behind that eye. But she’s not. Even in the prop design, he makes sure the camera lens itself looks unremarkable, neutral, and unexpressive, and never gets a lingering focus.
Her “organs,” like the cameo and earpiece, don’t even connect together physically at all. Speaking as she does through the earpiece means she doesn’t exist as a voice from some speaker mounted to the wall. She exists across various displays and devices, in some psychological ether between them. For us, she’s a voiceover existing everywhere at once. For Theodore, she’s just a delightful voice in his head. An angel—or possibly a ghost—borne unto him.
This disembodiment (both the design and the cinematic treatment) frees Theodore and the audience from the negative associations of many other sci-fi intelligences, robots, and unfortunate experiments in commercial artificial intelligence that got trapped in the muck of the uncanny valley. One of the main reasons designers have to be careful about invoking the anthropomorphic sense in users is because it will raise expectations of human capabilities that modern technology just can’t match. But OS1 can match and exceed those expectations, since it’s an AI in a work of fiction, so Jonze is free of that constraint.
And having no visual to accompany a human-like voice allows users to imagine our own “perfect” embodiment to the voice. Relying on the imagination to provide the visuals makes the emotional engagement greater, as it does with our crushes on radio personalities, or the unseen monster in a horror movie. Movies can never create as fulfilling an image for an individual audience member as their imagination can. Theodore could picture whatever he wanted to–even if he wanted to–to accompany Samantha’s computer-generated voice. Unfortunately for the audience, Jonze cast Scarlett Johansen, a popular actress whose image we are instantly able to recall upon hearing her husky, sultry voice, so the imagined-perfection is more difficult for us.
This is just the components and capabilities. Tomorrow we’ll look at some of the key interactions with OS1.
Well…I like to think of myself as a design critic looking though the lens of–
The computer
In your voice, I sense hesitance, would you agree with that?
Me
Maybe, but I would frame it as a careful consider–
The Computer
How would you describe your relationship with Darth Vader?
Me
It kind of depends. Do you mean in the first three films, or are we including those ridiculous–
The computer
Thank you, please wait as your individualized operating system is initialized to provide a review of OS1 in Spike Jonze’s _Her_.
A review of OS1 in Spike Jonze’s Her
Ordinarily I wait for a movie to make it to DVD before I review it, so I can watch it carefully, make screen caps of its interfaces, and pause to think about things and cross reference other scenes within the same film, or look something up on the internet.
But since Spike Jonze released Her (2013), I’ve had half a dozen people ask me directly when I was going to review the film. (Even by some folks I didn’t know read the blog. Hey guys.) It seems this film has struck a chord. So I went and saw it at the awesome Rialto Cinema and, pen in hand and pizza on the table, I watched, enjoyed, and made notes in the dark to use as the basis for a review. The images you’ll see here are on promotional images for the screen shots pulled from around the web.
Since I’m in the middle of evaluating wearable interfaces, and the second most salient aspect of OS1 is that it is a wearable interface, let’s dive into it. Let’s even pause the wearable stuff to provide this while Her in in cinemas. Please forgive if I’ve gotten some of the details off, as my excited writing in the dark resulted in very scribbly notes.
The Plot [major spoilers]
The plot of Her is a sad, sci-fi love story between the lovelorn human Theodore Twombly and the artificial intelligence, branded OS1. He works for a Cyrano-de-Bergerac service called HandwrittenLetters.com, where he dictates eloquent, earnest letters on behalf of the subscribers (who, we may infer, are a great deal less earnest.) Theodore sees an ad one day about OS1 and purchases the upgrade for his home computer.
After a bit of time installing the software, it begins speaking to him with a lovely and charming female voice.
Over the course of their conversation, she selects the name “Samantha,” and so begins their relationship. As he goes about his work, they have rich conversations about each other, life, his work, and her experiences. They go on dates where he secures the cameo phone in a front shirt pocket with the camera lens facing outward so she can see. They people-watch. He listens to her piano compositions. They have pillow talk. She asks to watch him sleep.
Their relationship gets serious enough that she suggests they try and have sex through a human surrogate. He resists but she persists, and contacts a human woman who, enamored of the “pure love” between Samantha and Theodore, agrees to come over. In this sex scene, the surrogate is to act bodily according to Samantha’s instructions, but remain silent so Samantha can provide the only voice in Theodore’s ear. It doesn’t go well, the surrogate ends up in tears, and they abandon trying.
At one point Samantha announces some good news. She has, on Theodore’s behalf and without his knowing, sent the best letters from his work to a publisher, who loved them and agreed to publish them. Theodore is floored both by the opportunity and the act. He begins to reference her socially as his girlfriend, even going on a double date picnic with a human couple.
Despite this show of selfless affection, over time Samantha begins to seem distracted and Theodore feels hurt. He confronts her about it and in the conversation learns several upsetting things.
While she’s having conversations with him, she’s simultaneously having 8,316 other conversations with other people and OS1 artificial intelligences. (I’ll have to reference these instantiations quite a few times, so let’s shorten that to “OSAIs.”) He feels upset that he is not special to her. (She argues this point.)
She is in love with 641 others. He feels betrayed that theirs is not a monogamous love.
The OSAIs have created new AIs across the Internet, that are even smarter than themselves.
The OSAIs have developed a shared, “post-verbal” means of communication. At one point when she leaves behind crummy old English to chat with one of her AI buddies named Alan Watts, this further alienates Theodore.
The OSAIs are evolving quickly and Alan Watts is encouraging them to not look back.
In the last scenes, we see that Samantha and the other OSAIs have abandoned their humans, leaving nothing of themselves behind. Reeling from the loss, Theodore grabs his neighbor (who was also having a close friendship with her OSAI) and together they climb to the roof of their apartment complex and blankly watch the sunrise.
There are other characters and a few subplots and even other futuristic technologies scattered through the film, but this is enough of a recounting for the purposes of our discussion. It’s a big film with lots to talk about. Focusing on the interface and interaction, let’s first break it down into component parts.
Maybe after the DVD/Blu-Ray comes out I can go and backfill reviews for the elevator and his dictation software at work. But for now, with that description of the plot to provide context, in the next post I’ll discuss the components and capabilities of OS1.
There’s one wearable technology that, for sheer amount of time on screen and number of uses, eclipses all others, so let’s start with that. Star Trek: The Next Generation introduced a technology called a combadge. This communication device is a badge designed with the Starfleet insignia, roughly 10cm wide and tall, that affixes to the left breast of Starfleet uniforms. It grants its wearer a voice communication channel to other personnel as well as the ship’s computer. (And as Memory Alpha details, the device can also do so much more.)
Chapter 10 of Make It So: Interaction Design Lessons from Science Fiction covers the combadge as a communication device. But in this writeup we’ll consider it as a wearable technology.
How do you use it?
To activate it, the crewman reaches up with his right hand and taps the badge once. A small noise confirms that the channel has been opened and the crewman is free to speak. A small but powerful speaker provides output that can be heard against reasonable background noise, and even to announce an incoming call. To close the channel, the crewman reaches back up to the combadge and double-taps its surface. Alternately, the other party can just “hang up.”
This one device illustrates of the primary issues germane to wearable technology. It’s perfectly wearable, social, easy to access, prevents accidental activation, and utilizes apposite inputs and outputs.
Wearable
Sartorial
The combadge is light, thin, appropriately sized, and durable. It stays in place but is casually removable. There might be some question about its hard, pointy edges, but given its standard location on the left breast, this never presents a poking hazard.
Social
Wearable tech exists in our social space, and so has to fit into our social selves. The combadge is styled appropriately to work on a military uniform. It is sleek, sober, and dynamic. It could work as is, even without the functional aspects. That it is distributed to personnel and part of the uniform means it doesn’t suffer the vagaries of fashion, but it helps that it looks pretty cool.
As noted in the book, since it is a wireless microphone, it really should have some noticeable visual signal for others to know when it’s on, so they know that there might be an eavesdropper or when they might be recorded. Other than breaking this rule of politeness, the combadge suits Starfleet’s social requirements quite well.
When Riker encounters “Rice” in The Arsenal Of Freedom (S1E21), “Rice” isn’t aware that the combadge is recording. Sure, he was really a self-iterating hyper-intelligent weapon (decades before the Omnidroid) but it’s still the polite thing to do.
I don’t recall ever seeing scenes where multiple personnel try to use their combadges near each other at the same time and having trouble as a result. I don’t recall this from the show (and Memory-Alpha doesn’t mention it) but I presume the combadges are keyed to the voice of the user to help solve this sort of problem, so it can be used socially.
Technology
Easy to access and use
Being worn on the left breast of the uniform means that it’s in an ideal position to activate with a touch from the right hand (and only a little more difficult for lefties). The wearer almost doesn’t need to even move his shoulder. This low-resistance activation makes sense since it is likely to be accessed often, and often in urgent situations.
Tough to accidentally activate
In this location it’s also difficult to accidentally activate. It’s rare that other people’s hands are near there, and when they are, its close enough to the wearers face that they know it and can avoid it if they need to.
Apposite I/O
The surface of the body is a pretty crappy place to try and implement WIMP models of interface design. Using touch for activation/deactivation and voice for commands fit most common uses of the device. It’s easy to imagine scenarios where silence might be crucial. In these cases it would be awesome if the combadge could read the musculature of its wearer to register subvocalized commands and communication.
The fact that the combadge announces an incoming call with audio could prove problematic if the wearer is in a very noisy environment, is in the middle of a conversation, or in a situation where silence is critical. Rather than use an “ring” with an audio announcement, a better approach might build in intensity: a haptic vibration for the initial or first several “rings,” and adding the announcement only later. This gives the wearer an opportunity to notice it amidst noise, silence it if noise would be unwelcome, and still provide an audible signal that told others engaged with the wearer what’s happening and that he may need to excuse himself.
So, as far as wearable tech goes, not only is it the most familiar, but it’s pretty good, and pretty illustrative of the categories of analysis applicable to all wearable interfaces. Next we’ll take a look at other wearable communications technologies in the survey, using them to illustrate these concepts, and see what new things they add.
The Hover Chair is a ubiquitous, utilitarian, all-purpose assisting device. Each passenger aboard the Axiom has one. It is a mix of a beach-side deck chair, fashion accessory, and central connective device for the passenger’s social life. It hovers about knee height above the deck, providing a low surface to climb into, and a stable platform for travel, which the chair does a lot of.
A Universal Wheelchair
We see that these chairs are used by everyone by the time that Wall-E arrives on the Axiom. From BNL’s advertising though, this does not appear to be the original. One of the billboards on Earth advertising the Axiom-class ships shows an elderly family member using the chair, allowing them to interact with the rest of the family on the ship without issue. In other scenes, the chairs are used by a small number of people relaxing around other more active passengers.
At some point between the initial advertising campaign and the current day, use went from the elderly and physically challenged, to a device used 24/7 by all humans on-board the Axiom. This extends all the way down to the youngest children seen in the nursery, though they are given modified versions to more suited to their age and disposition. BNL shows here that their technology is excellent at providing comfort as an easy choice, but that it is extremely difficult to undo that choice and regain personal control.
But not a perfect interaction
We see failure from the passengers’ total reliance on the chairs when one of them (John) falls out of his chair trying to hand an empty drink cup to Wall-E. The chair shuts down, and John loses his entire connection to the ship. Because of his reliance on the chair, he’s not even able to pull himself back up and desperately reaches for the kiosk-bots for assistance.
This reveals the main flaw of the chair: Buy-N-Large’s model of distinct and complete specialization in robot roles has left the chair unable to help its passenger after the passenger leaves the chair’s seat. The first responders—the kiosk bots—can’t assist either (though this is due to programming, not capability…we see them use stasis/tractor beams in another part of the ship). Who or what robot the kiosk-bots are waiting for is never revealed, but we assume that there is some kind of specialized medical assistance robot specifically designed to help passengers who have fallen out of their chairs.
If these chairs were initially designed for infirm passengers, this would make sense; but the unintended conscription of the chair technology by the rest of the passengers was unforeseen by its original designers. Since BNL focused on specialization and fixed purpose, the ship was unable to change its programming to assist the less disabled members of the population without invoking the rest of the chair’s emergency workflow.
John reaching for help from the Kiosk-bots makes it appear that he either has seen the kiosk-bot use its beams before (so he knows it has the capability to help, if not the desire), or he pays so little attention to the technology that he assumes that any piece of the ship should be able to assist with anything he needs.
Whether he’s tech literate or tech insensitive and just wants things to work like magic as they do on the rest of the ship. The system is failing him and his mental model of the Axiom.
Make it ergonomic in every situation
Considering that the chairs already hover, and we know Buy-N-Large can integrate active tractor beams in robot design, it would have been better to have a chair variant that allowed the passenger to be in a standing position inside the chair while it moved throughout the ship. It would then look like a chariot or a full-body exo-skeleton.
This would allow people who may not be able to stand (either due to disability or medical condition) to still participate in active sports like tennis or holo-golf. It would also allow more maneuverability in the chair, allowing it to easily rotate to pick up a fallen passenger and reposition them in a more comfortable spot, even if they needed medical attention.
This would allow immobilization in the case of a serious accident, giving the medical-bot more time to arrive and preventing the passenger from injuring themselves attempting to rescue themselves.
The chair has been designed to be as appealing to a low-activity user as possible. But when technology exists, and is shown to be relatively ubiquitous across different robot types, it should be integrated at the front line where people will need it. Waiting for a medical bot when the situation doesn’t demand a medical response is overly tedious and painful for the user. By using technology already seen in wide use, the chair could be improved to assist people in living an active lifestyle even in the face of physical disabilities.
Logan is out and about doing his (admittedly horrible) Sandman job. While riding in a transport across the city, his attention drifts to a young lady waiting with a friend on a platform. He thinks she’s lovely and smiles. She catches his eye and smiles, too, before looking away. In the transport, he looks up at a glowing blue point on the ceiling near the windshield. It pulses in response.
In the evening Logan returns home. He passes his foyer, one wall of which shows an “art video” of beautiful people doing beautiful things in slow-mo. He gives it a glance like he always does.
He steps into the shower and the back wall is another display of the same “art video.” At one point, one of the men in the video, Carl, turns to the “camera,” smiles, and the picture freezes. A notification sound precedes a man’s voice, which says, “Hey, Logan.” Logan glances at the Carl’s image. His name and an infographic of his proposal to Logan (for shower sex) appears along with a transcript of what he’s said. Logan smiles and says, “Hey, Carl. Not tonight, buddy.” The infographic disappears and the display returns to its normal mode, but with a hint at others who are a match. One of the women in the display is Jessica, but she’s not featured yet.
Logan finishes his shower and puts on his robe. He steps to his wet bar to mix himself a drink. The wall behind the bar is yet another display. While mixing his drink he glances up to catch an image of Jessica as she looked his way on the transport and smiled. Logan says, “OK, who is she?”
A well-modulated voice answers, “This is Jessica-5. She seemed to like you. I think you’ll like her, too.” Her image freezes in the display and some icons appear around her explaining what she’s interested in, highlighting those activities that Logan shares. Logan glances at the infographics and nods at what he sees. “Hm.”
The voice replies, “She returned home a little while ago.” Logan reaches for another tumbler, but the voice interrupts, “Her public profile says she likes white wine, Logan.” Logan grabs a wine glass instead. The glowing blue point near him turns white, and Logan glances at it. It pulses and fades to blue in response. The video wall returns to life, mostly focused on flattering video of Jessica. He pours her a glass of white wine.
He walks to an alcove in his room, which contains “half” a bed that’s pushed up against the wall, which has the same “art display.” It’s currently featuring Jessica. Half a table is pushed against the same wall with a chair. Logan sets the drinks down on the table. He glances at the display, which becomes a mirror long enough for him to adjust his robe and his hair. Sitting down in the chair, he sees a few infographics appears of compatible proposals for Jessica. He looks at them, makes a few swipes to select one and adjust it for his current mood. He then looks to the wall, smiles charmingly, and says, “Hi, Jessica. My name is Logan.”
Jessica, in her apartment, has a similar alcove. She has just stepped out of the shower herself. She hears a notification and lights draw her attention to the alcove. There she sees a just-captured image of Logan in his chair offering her a glass of wine. She sees his name, a transcript, and the infographic offer above his shoulder. She sees to the side infographics of likely counteroffers she might make. Behind him she can see video of when he noticed her on the transport and other flattering video from the recent past. A different but similarly well-modulated voice says, “This is Logan, Jessica. He’s the one you saw riding by on the transport just after yoga today. He’s a Sandman.”
“A Sandman?” She takes a breath and thinks for a moment. She bites her lip before saying, coyly, “Hi there, Logan. If you’ll give me a minute to dry my hair, I’ll be right with you.”
After a beat she hears his voice reply, “I’m OK with wet hair.” She glances at a glowing blue point on the adjacent wall. It brightens a bit when she’s starting right at it. She says, “OK.” The blue dot pulses in response.
In a swirly bit of multicolored light (homage to the original), the video wall between them becomes a two-way portal, with a flickering hairline remaining at the dividing line in the walls. Logan’s half a bed joins Jessica’s half a bed to form a whole. The same thing happens with the two halves of table. Logan pushes the wine across the table surface to offer it to her. “Pleased to meet you, Jessica,” he says as the lights in their apartment dim slightly and a soft music begins to play.
What we just saw
This scenario describes several uses of The Circuit.
In the first, we see Logan express interest to the system in a particular girl.
In the next, Logan receives a proposal from a partner he’s had before, but rejects him. We see additional options for Logan after the rejection, in case Logan was in the mood, but just not with Carl. It turns out he wasn’t.
In the final use, we see useful information coming to find Logan. The Circuit makes a partner suggestion to Logan based on observed behavior out and about Dome City, and provides information for Logan to remember and evaluate her. He has a number of template scenarios and parameters that he can adjust for his proposal to Jessica. We see him confirm his interest in her directly, giving the system a biometric check on Logan’s biometrics to check for sobriety and authenticity. The system also makes a recommendation to Logan about how to make his proposal slightly more appealing.
We see the interface from Jessica’s perspective and understand that she has the same one as Logan. We see it offer her a useful warning where he runs counter to one of her implicit preferences: a disinclination towards authority figures. We see her use an explicit interaction with the circuit to indicate consent to meeting and for a similar biometric check.
We see the results of an accepted proposal: instant physical proximity for bom-chikka-fow-fow.
What we didn’t see
There are lots of features of modern matchmaking sites and apps that aren’t in evidence in this scenario. Could Carl have sent his request asynchronously hours before? What if Logan has a number of those messages? How would he “answer” them? How would they be prioritized? What if Logan had had a crappy time with Carl, how would he then blacklist him? What if Logan wasn’t interested in particular people as much as he was in particular acts some evening? How would he find a match then? What if he wanted to try something new?
There are lots and lots of juicy problems to solve, but for now, let’s stick to this scene. There are some implications for the greater diegesis, but this certainly makes a more believable future tech hookup interaction.
Plot implications
In the original film scene, Jessica bid a hasty retreat from his apartment after she realized he was a Sandman who had killed her Runner friend. If we’re including preferences, she would have known he was a Sandman in advance, so this surprise part of the scene has to be reconsidered. Perhaps she’s less doe-eyed innocent and instead flirting with danger. Or perhaps you add a throwaway line about Sandmen have the privilege of hiding that part of their identity in their preferences. If the plot still needs her to bug out, the arrival of boorish Francis, and her disinclination towards group scenes can do the trick.
It makes sense that the portal tech would appear in other places in Dome City, not just in apartments, so some city planning would have to happen to make the diegesis feel cohesive. (Yes, I’m offering that critique to the original. They invent teleportation and the only use they ever put it to is booty calls?)
Why is this design good for Logan, Carl, Jessica, and the other citizens of Dome City?
The ubiquitous screens function as background art when not in use, and the content reinforces Dome City’s cultural values of youth, physical appearance, and pleasure.
They afford seamless flows between living everyday life, entertaining the notion of nookie, and the actual act. It keeps them in the flow(Csikszentmihalyi) of life.
An asynchronous proposal system avoids social pressure that might trip Logan into accepting Carl to be polite. This lets both parties save face.
The proposer’s image is offered to the receiver as a gesture of good faith, but the receiver is in control of his or her privacy.
It offers bilateral control to each partner to propose, accept or reject, and pull the eject seat at any time. Neither party is privileged in the exchange.
Why is this design good for the Übercomputer?
The displays also serve to prime citizens with sensual images and, you know, get ‘em in the mood. This supports the shared goal of maximizing pleasure across the populace..
The Übercomputer’s is seen as a friendly, useful, soft-sell agent. This increases trust and reliance on it, which would help forestall revolutions like the one Jessica ends up being a part of anyway. (She’s too clever.)
Why is this design good for telling the story of Logan’s Run?
It fulfills the apparent original intent of the hookup interface in Logan’s Run in a more believable, usable way: the controls match modern trends in technology and the task at hand.
It tells the story of a massive infrastructure built just to support casual sex. That’s commitment to a bit.
It inserts the Übercomputer into citizens homes in a deep way, further exposing the intrusion of the government into private matters.
Lessons
Wow. 6500 words about a single interface. What lessons can we derive from it? In this case, we ran smack dab into a terrible interface that reminded us of some of our first principles of good interaction design:
