Biff(2015) pays for his taxi ride to the McFly household with his thumbprint. When the ride ends, a synthesized voice gives the price one-seven-four-point-five-zero. The taxi driver presents him with a book-sized device with the price at the top on a red 7-segment LED display. Biff presses his thumb on a reader at the bottom that glows white as it scans. When the payment is verified, the thumbprint reader and the price go dark as a sound plays like a register.
For due diligence, let me restate: multimodal biometric or multifactor authentication is more secure.
Dr. Brown gives Marty some 21st century clothes in order to blend in. The first of these items are shoes. Marty is surprised to see no laces. To activate them, he pushes his foot into the shoe. When his heel makes contact, the main strap constricts to hold his heel in place. Then the laces constrict to hold the ball of the heel down. Finally, the tongue of the shoe and the Nike logo glow.
Yep. Perfect. The activation is natural to the act of putting on the device. The glow acts as a status indicator and symbol. No wonder everyone wanted them.
After the gravitic distortion is discovered, Barcalow flips a toggle switch upwards with his thumb. As Ibanez confirms that “Gravity is 225 and rising,” the light on the bridge turns red, and Barcalow turns to a monitor.
The monitor (seen above) features a video window in the top center. Along the left side of the screen 11 random numbers report the COMM STATS INTERSHIP. Along the right side of the screen 11 other random numbers report the COMM STATS INTRASHIP. Beneath the video some purple bars slide in and out from a central column of red rectangles. One of these rectangles is bright yellow. Beneath that a section reports SCANNING FREQUENCIES as 21 three-character strings, some of which are highlighted as red. At the bottom of the screen blue and yellow-green smears race back and forth across a rectangle. Everything is in Starship Troopers‘ signature saturated colors and a block font like Microgramma or Eurostile.
These details are almost immediately obscured, as Deladier looks up from her laptop (looking presciently like a modern Macbook Air with its aluminum casing) to look at the video monitor to demand a “Report,” and the video grows larger to fill the screen.
Here the snarky description must pause for some analysis.
Analysis
The red alert mechanism is actually pretty good. Both the placement of its switch at shoulder level and the fact that it must be flipped up help prevent against accidental activation. The fact that it’s a toggle switch means it can be undone with ease if necessary. The red light immediately provides feedback to everyone on the bridge (and throughout the ship?) that the system has gone into a red alert. No other action is necessary to alert the person who needs to be informed, i.e. the Captain. The only other improvement might be a klaxon warning to alert others who are sleeping, but it’s entirely possible that very thing is happening elsewhere on the ship, and the bridge is spared that distraction. So full marks.
The user interface on the monitor seems pretty crappy though. If someone is meant to monitor COMM STATS—intership or intraship—I cannot imagine how a column of undifferentiated numbers helps. A waveform would be more useful to track activity across a spectrum. Something. Anything other than a stack of numbers that are hard to read and interpret.
The SCANNING FREQUENCIES is similarly useless. Sure, it’s clear that the ship’s systems are scanning those frequencies, but the three-character strings require crew to memorize what those mean. If those frequencies are defined—as you imagine they must be to be at all useful as static variables—then you can remove the cognitive weight of having to memorize the differences between JL5 and LQ7 by giving them actual names, and only displaying the ones that have activity on them, and what that activity means. Does someone need to listen in? Shouldn’t that task be apparent? And why would that need to be shown generally to the bridge, rather than to a communications officer? And I’m not sure what those purple squiggles mean. It’s nice that they’re animated I guess, but if they’re meant to help the user monitor some variable, they’re too limited. Like the sickbay display on the original Star Trek, knowing the current state is likely not as useful as knowing how the information is trending over time. (See page 261 for more details on this.) So trendlines would be better here. The little sweeping candy colored smears are actually okay, though, presuming that it’s showing that the system is successfully sweeping all frequencies for additional signal. Perhaps a bit distracting, but easy to habituate.
It’s nice that the video screen fills the screen to match the needs of the communicators. But as with so many other sci-fi video calls, no effort is made to explain where the camera is on this thing. Somehow they can just look at the eyes of the other person on the monitor, and it works. This feels natural to the actors, looks natural to the audience, and would be natural in real life, but until we can figure out how to embed a camera within a screen, this can’t work this way, and we’re stuck with the gaze monitoring problem raised in the Volumetric Projection chapter of the book with the Darth Vader example.
So, all in all, this interface is mostly terrible until it becomes just a videophone. And even then there are questions.
Snarky description continues
Picking up the description where I left off, after the Captain demands a report, Barcalow tells her quickly “Captain, we’re in the path of an unidentified object heading toward us at high speed.” Ibanez then looks down at her monitor at the gravity well animation, to remark that the “Profile suggests an asteroid, ma’am.” You know, just before looking out the window.
Honestly, that’s one of the funniest two-second sequences in the whole movie.
Well…I like to think of myself as a design critic looking though the lens of–
The computer
In your voice, I sense hesitance, would you agree with that?
Me
Maybe, but I would frame it as a careful consider–
The Computer
How would you describe your relationship with Darth Vader?
Me
It kind of depends. Do you mean in the first three films, or are we including those ridiculous–
The computer
Thank you, please wait as your individualized operating system is initialized to provide a review of OS1 in Spike Jonze’s _Her_.
A review of OS1 in Spike Jonze’s Her
Ordinarily I wait for a movie to make it to DVD before I review it, so I can watch it carefully, make screen caps of its interfaces, and pause to think about things and cross reference other scenes within the same film, or look something up on the internet.
But since Spike Jonze released Her (2013), I’ve had half a dozen people ask me directly when I was going to review the film. (Even by some folks I didn’t know read the blog. Hey guys.) It seems this film has struck a chord. So I went and saw it at the awesome Rialto Cinema and, pen in hand and pizza on the table, I watched, enjoyed, and made notes in the dark to use as the basis for a review. The images you’ll see here are on promotional images for the screen shots pulled from around the web.
Since I’m in the middle of evaluating wearable interfaces, and the second most salient aspect of OS1 is that it is a wearable interface, let’s dive into it. Let’s even pause the wearable stuff to provide this while Her in in cinemas. Please forgive if I’ve gotten some of the details off, as my excited writing in the dark resulted in very scribbly notes.
The Plot [major spoilers]
The plot of Her is a sad, sci-fi love story between the lovelorn human Theodore Twombly and the artificial intelligence, branded OS1. He works for a Cyrano-de-Bergerac service called HandwrittenLetters.com, where he dictates eloquent, earnest letters on behalf of the subscribers (who, we may infer, are a great deal less earnest.) Theodore sees an ad one day about OS1 and purchases the upgrade for his home computer.
After a bit of time installing the software, it begins speaking to him with a lovely and charming female voice.
Over the course of their conversation, she selects the name “Samantha,” and so begins their relationship. As he goes about his work, they have rich conversations about each other, life, his work, and her experiences. They go on dates where he secures the cameo phone in a front shirt pocket with the camera lens facing outward so she can see. They people-watch. He listens to her piano compositions. They have pillow talk. She asks to watch him sleep.
Their relationship gets serious enough that she suggests they try and have sex through a human surrogate. He resists but she persists, and contacts a human woman who, enamored of the “pure love” between Samantha and Theodore, agrees to come over. In this sex scene, the surrogate is to act bodily according to Samantha’s instructions, but remain silent so Samantha can provide the only voice in Theodore’s ear. It doesn’t go well, the surrogate ends up in tears, and they abandon trying.
At one point Samantha announces some good news. She has, on Theodore’s behalf and without his knowing, sent the best letters from his work to a publisher, who loved them and agreed to publish them. Theodore is floored both by the opportunity and the act. He begins to reference her socially as his girlfriend, even going on a double date picnic with a human couple.
Despite this show of selfless affection, over time Samantha begins to seem distracted and Theodore feels hurt. He confronts her about it and in the conversation learns several upsetting things.
While she’s having conversations with him, she’s simultaneously having 8,316 other conversations with other people and OS1 artificial intelligences. (I’ll have to reference these instantiations quite a few times, so let’s shorten that to “OSAIs.”) He feels upset that he is not special to her. (She argues this point.)
She is in love with 641 others. He feels betrayed that theirs is not a monogamous love.
The OSAIs have created new AIs across the Internet, that are even smarter than themselves.
The OSAIs have developed a shared, “post-verbal” means of communication. At one point when she leaves behind crummy old English to chat with one of her AI buddies named Alan Watts, this further alienates Theodore.
The OSAIs are evolving quickly and Alan Watts is encouraging them to not look back.
In the last scenes, we see that Samantha and the other OSAIs have abandoned their humans, leaving nothing of themselves behind. Reeling from the loss, Theodore grabs his neighbor (who was also having a close friendship with her OSAI) and together they climb to the roof of their apartment complex and blankly watch the sunrise.
There are other characters and a few subplots and even other futuristic technologies scattered through the film, but this is enough of a recounting for the purposes of our discussion. It’s a big film with lots to talk about. Focusing on the interface and interaction, let’s first break it down into component parts.
Maybe after the DVD/Blu-Ray comes out I can go and backfill reviews for the elevator and his dictation software at work. But for now, with that description of the plot to provide context, in the next post I’ll discuss the components and capabilities of OS1.
There’s one wearable technology that, for sheer amount of time on screen and number of uses, eclipses all others, so let’s start with that. Star Trek: The Next Generation introduced a technology called a combadge. This communication device is a badge designed with the Starfleet insignia, roughly 10cm wide and tall, that affixes to the left breast of Starfleet uniforms. It grants its wearer a voice communication channel to other personnel as well as the ship’s computer. (And as Memory Alpha details, the device can also do so much more.)
Chapter 10 of Make It So: Interaction Design Lessons from Science Fiction covers the combadge as a communication device. But in this writeup we’ll consider it as a wearable technology.
How do you use it?
To activate it, the crewman reaches up with his right hand and taps the badge once. A small noise confirms that the channel has been opened and the crewman is free to speak. A small but powerful speaker provides output that can be heard against reasonable background noise, and even to announce an incoming call. To close the channel, the crewman reaches back up to the combadge and double-taps its surface. Alternately, the other party can just “hang up.”
This one device illustrates of the primary issues germane to wearable technology. It’s perfectly wearable, social, easy to access, prevents accidental activation, and utilizes apposite inputs and outputs.
Wearable
Sartorial
The combadge is light, thin, appropriately sized, and durable. It stays in place but is casually removable. There might be some question about its hard, pointy edges, but given its standard location on the left breast, this never presents a poking hazard.
Social
Wearable tech exists in our social space, and so has to fit into our social selves. The combadge is styled appropriately to work on a military uniform. It is sleek, sober, and dynamic. It could work as is, even without the functional aspects. That it is distributed to personnel and part of the uniform means it doesn’t suffer the vagaries of fashion, but it helps that it looks pretty cool.
As noted in the book, since it is a wireless microphone, it really should have some noticeable visual signal for others to know when it’s on, so they know that there might be an eavesdropper or when they might be recorded. Other than breaking this rule of politeness, the combadge suits Starfleet’s social requirements quite well.
When Riker encounters “Rice” in The Arsenal Of Freedom (S1E21), “Rice” isn’t aware that the combadge is recording. Sure, he was really a self-iterating hyper-intelligent weapon (decades before the Omnidroid) but it’s still the polite thing to do.
I don’t recall ever seeing scenes where multiple personnel try to use their combadges near each other at the same time and having trouble as a result. I don’t recall this from the show (and Memory-Alpha doesn’t mention it) but I presume the combadges are keyed to the voice of the user to help solve this sort of problem, so it can be used socially.
Technology
Easy to access and use
Being worn on the left breast of the uniform means that it’s in an ideal position to activate with a touch from the right hand (and only a little more difficult for lefties). The wearer almost doesn’t need to even move his shoulder. This low-resistance activation makes sense since it is likely to be accessed often, and often in urgent situations.
Tough to accidentally activate
In this location it’s also difficult to accidentally activate. It’s rare that other people’s hands are near there, and when they are, its close enough to the wearers face that they know it and can avoid it if they need to.
Apposite I/O
The surface of the body is a pretty crappy place to try and implement WIMP models of interface design. Using touch for activation/deactivation and voice for commands fit most common uses of the device. It’s easy to imagine scenarios where silence might be crucial. In these cases it would be awesome if the combadge could read the musculature of its wearer to register subvocalized commands and communication.
The fact that the combadge announces an incoming call with audio could prove problematic if the wearer is in a very noisy environment, is in the middle of a conversation, or in a situation where silence is critical. Rather than use an “ring” with an audio announcement, a better approach might build in intensity: a haptic vibration for the initial or first several “rings,” and adding the announcement only later. This gives the wearer an opportunity to notice it amidst noise, silence it if noise would be unwelcome, and still provide an audible signal that told others engaged with the wearer what’s happening and that he may need to excuse himself.
So, as far as wearable tech goes, not only is it the most familiar, but it’s pretty good, and pretty illustrative of the categories of analysis applicable to all wearable interfaces. Next we’ll take a look at other wearable communications technologies in the survey, using them to illustrate these concepts, and see what new things they add.
One of the most impressive interfaces seen in the film is Joh’s wall-mounted videophone. It is a marvel of special effects for 1927, and an ideal example that no matter how far sci-fi wants to look into the future, it must base its interfaces on the paradigms familiar to the audiences. Note in Joh’s use of the interface how the videophone is an awkward blend of early 20th century technology metaphors.
The videophone is a large device, easily as tall as Joh himself, mounted to the wall of his office. At its center is a large vertically-oriented video screen that is angled upwards for easy downward viewing. To the right and left of the screen are large tuning dials. A series of knobs and controls sit below the dials.
Joh checks the recent activity of the video phone.
When he first approaches the device, he checks the tickertape dangling from an overhung box on the left. Not seeing anything of interest, he drops the paper and approaches the screen.
Joh tunes in the channel he needs to speak to Grot.
Reaching up to the right dial, he turns its hand counterclockwise from pointing at the number 10 to the number 6. Then he turns the left hand dial to 4, and the screen comes to life. It first displays the legend HM 2 at the top. Some video appears below this, but rather than a clear feed of a single camera, it is a shifting blend of different cameras. Joh must fiddle with a few controls to clear the reception.
This moment seems quite strange to viewers familiar with modern video technology, since their experience is rooted in VHF broadcast, cable television, or online video. With these technology metaphors, channels are discrete. But it is important to recall that television was not popularized at the time, and the media metaphor most familiar to that audience would be radio, which users do in fact have to tune to get a clear signal.
Joh picks up the phone and calls Grot.
Joh verifies that hes seeing the right channel visually, by seeing Grots nervous pacing in camera view. Confident that he’s calling the right place, Joh picks up a telephone handset from the device, and reaches across to repeatedly press a button on the right. In response, the light bulbs on Grot’s videophone begin to blink and (presumably) make a sound.
Joh tells Grot to destroy the Heart Machine.
Grot rushes to his device, looks into the screen and lifts his handset. The two have a conversation, each looking directly at the face on the screen.
This moment is another telling one. Lang was familiar with cameras, and could have had his actors talking to a lens. But instead he had them do what felt right and would make sense to the audience, i.e. talking to the other person, not the machine. In this way Lang is involved in bodystorming the right feel of technology, and in so doing is setting expectations for the way the real technology—should it ever get here—should work.
His command issued, Joh hangs up on an incredulous Grot.
A final note on the interaction is that, to end the call, Joh returns the handset to its resting position, and, much like a telephone, this ends the call for both parties.
This seems to us like an overcomplicated mash-up of technology metaphors: telegraph, film, radio, and telephone. Of course hindsight is 20/20, but it’s not that Lang lacked the vision. He easily could have made the device more magic, by omitting the telephone handset and have Joh speak directly to the image of Grot. But Lang was not a technologist. He was a filmmaker, and needed to take his audience on the journey with him. He spoke to them in their shared language, using understandable cues to the individual components that, when added together add up to the something new that is one of the delights and promises of science fiction.
Dr. Brown gives Marty some 21st century clothes in order to blend in. The first of these items are shoes. Marty is surprised to see no laces. To activate them, he pushes his foot into the shoe. When his heel makes contact, the main strap constricts to hold his heel in place. Then the laces constrict to hold the ball of the heel down. Finally, the tongue of the shoe and the Nike logo glow.
