An Interview with Mark Coleran

 

In homage to the wrap of Children of Men, this post I’m sharing an interview with Mark Coleran, a sci-fi interface designer who worked on the film. He also coined the term FUI, which is no small feat. He’s had a fascinating trajectory from FUI, to real world design here in the Bay Area, and very soon, back to FUI again. Or maybe games.

I’d interviewed Mark way back in 2011 for a segment of the Make It So book that got edited out of the final book, so it’s great to be able to talk to him again for a forum where I know it will be published, scifiinterfaces.com.

This interview has been edited for clarity and length.

Tell us a bit about yourself.

So obviously my background is in sci-fi interfaces, the movies. I spent around 10 years doing that from 1997 to 2007. Worked on a variety of projects ranging from the first one, which was Tomb Raider, through to finishing off the last Bourne film, Bourne Ultimatum.

The Bourne Ultimatum, from Mark’s online portfolio, https://www.behance.net/markcoleran

My experience of working in films has been coming at it from the angle of loving the technology, loving the way machines work. And trying to expose it, to make it quite genuine. That’s what I got a name for in the industry was to try and create a more realistic side of interfaces.

Why is it hard to create FUI that would also work in the real world?

It’s because most people have no idea what an interface is, or what it’s supposed to be. From the person watching, for the actor using, the person designing, the person writing, the person directing, they don’t really know why it is there. This is the fundamental problem of the idea of sci-fi interfaces, they’re not interfaces. What they are are plot visualizations. They’re there to illustrate, or demonstrate something happening, or something that has happened. Or connect two people together in space.

So the work of the FUI designer is, working quickly, to fulfill the script, the plot point. Secondarily you consider the style of set design, context, story segment, things like that. That’s not the way things get made in the real world. Film UX and film UI are very much two separate things.

Consider this. If we made things that worked for actors to use on set,  the second that actor starts using something, they stop performing, they stop acting. So we can’t make something they actually use during filming. We have to play man behind the curtain, controlling the interface, matching their performance. That allows us to tell the actors, “Do not think about it, just do it. Just do your acting.” So when you see incoherent mashing on the keys and senseless clicking or mouse movement, it’s because we told them to do that.

Imagine how dull it would be to watch a film of a real person trying to figure out real software. There’s a line of realism you can’t cross. You don’t want a genuine database lookup of a police suspect. It’s a user experience problem wrapped in a user experience problem.

Let’s talk specifically about Children of Men. It’s now 10 years old. What do you think of when you look back on that work?

It was a really brief job, I only spent two weeks on the entire thing. It was a subcontract by a company called the Foreign Office. And the lead director was Frederick Norbeck, I think. So their commission was to design all of the advertisements in the film.

They did a lot of the backgrounding and the signage and they brought me in for the technology side of it, and also to create kind of brief world guide. For that I would just draw a timeline. Here’s what it’s like now, here’s where this unknown fertility event happens in five, six years time, and then the story in the film happens 20 years after that. Then I asked, “Okay, what is it like there? What were the systems like?”

As a result of the fertility event, all major technological advancement stops, so half the job was looking at just roughly where we’re gonna be in a couple of years and predicting how that technology will decay.

That’s why the paper has moving images, but they’ve got black lines and those things. It’s decaying.

In addition to the world book, I did a music player for the Forest House. I did all the office computers at the beginning. The signage for the Tate. And the game Kubris.

The step-through security gate & intuitive design

I liked the signage we did just for the step-through security gate. There’s a level of paranoia in that shot. On the side are four icons, like, “Radiation, weapons, explosives, biohazard.” Tiny, hard even to notice, but they tell of the scope of the problems they’re facing. Or expecting to face. 

It gets at a larger issue with a lot of these things. When you and I first spoke [for the book Make It So], I was kind of dismissive about a lot of the background of what we do, and what I do. It’s just like, stuff, I’d said. Make It So made me stop and ask, “What am I doing in my design?” There’s not a lot of time in any of these jobs. You have to work with your intuitive sense of design, with your vision based on your experience. Everything you’ve ever played, everything you’ve ever watched. It all has to go in. You have time to reflect later.

The Kubris Game

There’s a great lack of reflection at the front edge really. With the Kubris game all I got was, “It’s a game in a cube.”

“Okay,” I thought, “It’s space, let’s have him manipulate the space of the cube.” Maybe he’s pulling it, and it’s tumbling. But why is it tumbling? “Okay, let’s have pieces sliding down and if they go too far they’ll slide off the face, so he has to keep all these more and more pieces moving, sliding.” At a certain point you feel, “Oh that could be an interesting little game.” And it would play well in the scene.

It took me two days to go from that idea to having it on screen.

What made that project particularly challenging and unique?

The vast majority of films are just reflections of what we have right now, but Children of Men actually felt like it was trying to step ahead and show how things might really be. The temptation in a lot of technology to do the shiny thing, and this world is anything but shiny. So how does this technology reflect this real environment. But in this film, the interfaces aren’t the focus of any scene. It’s all there, but it’s just low-key texture.

What’s the worst FUI trope?

I want to say translucent screens, but I see why that’s become a trope. Having them transparent makes them feel like they’re part of the scene, rather than an object on a desk. Plus you get to see the actor’s faces. There’s an interesting connection to your crossover concept here [that is, that sci-fi and the real world mutually influence each other, see the talk about it at the O’Reilly recording here, or the post about transparent screens]. About 2–3 years ago I started to see translucent screens on the market, and I suspect the idea to create them came from sci-fi. The problem is, none of them could do true black, so they never really looked right.

No, a true trope vortex are spinning 3D globes and “flying” to information. I remember the original Ghost in the Shell. When Togusa looks at section 9 security, he says, “Show me something.” In response you it takes like three seconds for this building to spin just to show him the thing he just asked for. I’m like, “Uh…WHY?” [laughter] And FUI designers just keep going back to it, building on it, making it worse every time. It’s like it’s faster, and faster, and faster, and it just breaks apart.

GitS-Sec9_security-04

Going from FUI to real-world design and back again

I was called to do motion graphics and some interface work on…I’m not even gonna say which film it was. But I worked with through one of the most brilliant crews you can imagine. And despite all our incredible work, this film just…sucked, really bad. And I recall thinking, “It doesn’t matter who you are and what you do on a movie, you have no control whatsoever as to the outcome.”

So I thought I’d shift to work in the real world. Did some stuff in Canada, some really progressive stuff about file management and projects, how we visualize those things and work on them. Then I came to Silicon Valley, doing more work here, only to learn the lie of Silicon Valley: Designers believe they’re doing something positive and good. Really, you’re just subsuming whatever vision you have to somebody else’s idea of minimum viable product. Which in itself is fundamentally wrong, they should be minimum valuable product.

There’s also the horrible trade off between being an in-house designer, and having your ideas ignored by the higher ups, or being an external consultant, and having a very limited quality assurance in the execution of your ideas.

Hilariously, I once worked in-house on a TV project (again, I won’t mention names) and the team had some beautiful ideas. We presented them, and while we were waiting for the response of the higher ups, one of them decided “We need to get some external company to do this.” So they contacted an external firm, and two days later, I get a phone call from that company asking if I’m available to do the work as a subcontractor. It was very surreal. In reflecting on this I realized that I had a lot more influence on technology trends when I was working in the movies.

So now I’m heading back to that world.

What are your favorite Sci-Fi interfaces? Either that you or somebody else has created.

There’s a couple of them, one was the comlock from Space 1999. I loved the simplicity of that idea. It was a small thing, but it had an actual television screen, two inches wide. The characters pick it up off their belts, and look into it. So it all looks like they’re doing a kind of video karaoke. The best thing was it was all working display technology. They did some fancy camera work to hide the wires to the airstream next door with all the equipment that made these little things work. It was Graham Car’s work, and it was phenomenal.

Secondarily, I’d say the lap gun lasers from Aliens. [Seen in director’s cut, or unedited versions of the movie.] It’s just a laptop with a countdown of remaining ammunition. It was a simple, beautiful way of telling a piece of story. It was so elegantly done, and yet such attention to it. I really, really liked that.

One thing that stood in my mind recently, was Arrival. All the mundane use of technology was really nice. It’s still a background, a way characters are trying to tackle the problem, but it shows how they think. Like on the tablets, you draw or reselect pieces, build a structure from them. Beautifully done.

Then a surprising one is Assassin’s Creed.They changed the interface from the games. Look for the screens in the background, which are beautiful. Really different than a lot of people have done. Black and white. Very subtle in a lot of ways. There were all those little squares, doing things, very busy. It almost feels like it could’ve suddenly made something. It’s elegantly done.

If you could have any Sci-Fi tech made real, what would it be?

I want The Hitchhiker’s Guide to the Galaxy. I love the idea of having a guide for everything. A snarky guide for everything. It would probably get you into trouble, but at least make life interesting. Google Maps is just too damn good at what it does, it’s like, you need some variety in life. It’s the idea of an imperfect piece of technology could make your life interesting, or at least fun.

Report Card: Children of Men

Sci-fi UI report card for Children of Men: Sci A, Fi A, Interfaces A. Overall A+ (12 of 12), Blockbuster.

Depending on how you count, there are only 9 interfaces in Children of Men. This makes sense because it’s not one of those Gee-Whiz-Can-You-Believe-the-Future technofests like Forbidden Planet or Minority Report. Children of Men is a social story about the hopelessness of a world without children, so the small number of interfaces—and even the way they are underplayed—is wholly appropriate to the theme. Given such a small number, you would not expect them to be as spectacular as they are. Or maybe you would. I don’t know how you roll.

Sci A 4 of 4How believable are the interfaces?

The interfaces are wholly believable, given the diegesis. Technology is focused on security, transportation, and distracting entertainment, which is exactly what you’d expect. Nothing breaks physics or reason.

The only ding is that Quietus could have included some nod to its reimbursement promise, and that’s so minor it only reveals itself as a problem after deep consideration. It doesn’t break the flow of the film.

Fi A 4 of 4How well do the interfaces inform the narrative of the story?

All of the interfaces point back in some way to the world that created them and help move the story along. Security is everywhere. Jasper cobbles together technology to help his resistance. Suicide is a government-sanctioned option.

Interfaces A 4 of 4How well do the interfaces equip the characters to achieve their goals?

  • Luke’s HUD is a little slow, considering that its job is to help avoid collisions.
  • Jasper’s Home Alarm could do more to help its occupants respond effectively to the alarm.
  • The Music Player isn’t very readable at a distance.

These are the main three issues that mar an otherwise very well-considered set of interfaces and technologies.

Overall A+ 12 of 12Blockbuster

It’s rare that a film’s interfaces get a full blockbuster rating on this site. The only other one at the time of publication is The Fifth Element. And while I take pains to rate the interfaces as distinct from the movie, I’m pleased when such a brilliant (yet, ironically, dark) film includes brilliant interfaces as well.

Full series, in order: Children of Men (2007)

IMDB: https://www.imdb.com/title/tt0206634/

Ministry of Art detector gate

ChildrenofMen-Artgate-01

Jumping back in the film a bit, we’re going to visit the Ministry of Art. When Theo goes there to visit his brother, after the car pulls to the front of the secured building, Theo steps out and walks toward a metal-detector gate.

Its quite high, about 3 meters tall. The height helps to reinforce the notion that this is a public space.

  1. This principle, that short ceilings are personal, and high ceilings are public, is I believe a well-established one in architectural design. Read the Alexandrian pattern if you’d like to read more about it.
  2. Is it a public space? It is, since it’s a Ministry. But it isn’t, since he joins his brother in what looks like a rich person’s private dining room. I was always a bit confused by what this place was meant to be. Perhaps owning to The Dark Times, Nigel has cited Minister rights and cordoned off part of the Tate Modern to live in. If anyone can explain this, please speak up.
  3. On the downside, the height makes the text more out of sight and harder to read by the people meant to be reading it.

The distance is balanced by the motion graphics of the translucent sign atop the gate. Animated red graphics point the direction of ingress, show a security stripe pattern, and provide text instructions.

Motion is a very strong attention-getting signal, and combined with the red colors, does all the attention-getting that the height risks. But even that’s not a critical issue, as there is of course a guard standing by to ensure his understanding and compliance.

ChildrenofMen-Artgate-05

Note that there is no interaction here (which is the usual filter for this blog), but since I’m publishing an interview with the designer of this and the Kubris interface soon, I thought I’d give it a quick nod.

Jasper’s Music Player

ChildrenofMen-player03

After Jasper tells a white lie to Theo, Miriam, and Kee to get them to escape the advancing gang of Fishes, he returns indoors. To set a mood, he picks up a remote control and presses a button on it while pointing it at a display.

ChildrenofMen-player02

He watches a small transparent square that rests atop some things in a nook. (It’s that decimeter-square, purplish thing on the left of the image, just under the lampshade.) The display initially shows an album queue, with thumbnails of the album covers and two bright words, unreadably small. In response to his button press, the thumbnail for Franco Battiato’s album FLEURs slides from the right to the left. A full song list for the album appears beneath the thumbnail. Then track two, the cover of Ruby Tuesday, begins to play. A small thumbnail to the right of the album cover appears, featuring some white text on a dark background and a cycling, animated border. Theo puts the remote control down, picks up the Quietus box, and walks over to Janice. *sniff*

This small bit of speculative consumer electronics gets around 17 seconds of screen time, but we see enough to consider the design. 

Persistent display

One very nice thing about it is that it is persistently visible. As Marshall McLuhan famously noted, we are simply not equipped with earlids. This means that when music is playing in a space, you can’t really just turn away from it to stop listening. You’ll still hear it. In UX parlance, sound is non-modal.

Yet with digital music players, the visual displays that tell you about what’s being played, or the related interfaces that help you know what you can do with the music are often hidden behind modes. Want to know what that song you can’t stop hearing is? Find your device, wake it up, enter a password, find the app, and even then you may have to root around to find the software to find what you’re looking for.

But a persistent object means that non-modal sound is accompanied by (mostly) non-modal visuals. This little box is always somewhere, glowing, and telling you what’s playing, what just played, and what’s next.

Remote control

Finding the remote is a different problem, of course, and if your household is like my household, it is a thing which seems to want to be lost. To keep that non-modality of sound matched by the controls, it would be better to have the device or the environment know when Jasper is looking at the display, and enable loose gestural or voice controls to control it.

Imagine the scene if he grabs the Quietus box, looks up to the display, and says, “Play…” then pause while he considers his options, and says “…‘Ruby Tuesday’…the Battiato one.” We would have known that his selection has deep personal meaning. If Cuarón wanted to convey that this moment has been planned for a while, Jasper could even have said, “Play her goodbye song.”

Visual layout

The visual design of the display is, like most of the technology, meant to be a peripheral thing, accepting attention but not asking for it. In this sense it works. The text is so small the audience is not tempted to read it. The thumbnails are so small it is only if you already knew the music that it would refresh your memory. But if this was a real product meant to live in the home, I would redesign the display to be usable at the 3–6 meter distance, which would require vastly reducing the number of elements, increasing their size, and perhaps overlaying text on image.

ChildrenofMen-player03

Jasper’s home alarm

When Theo, Kee, and Miriam flee the murderous Fishes, they take refuge in Jasper’s home for the night. They are awoken in the morning by Jasper’s sentry system.

ChildrenofMen_Jasper_alarm

A loud cacophonous alarm sounds, made up of what sounds like recorded dog barks, bells clanging, and someone banging a stick on a metal trash can lid. Jasper explains to everyone in the house that “It’s the alarm! Someone’s breaking in!”

They gather around a computer screen with large speakers on either side. The screen shows four video feeds labeled ROAD A, FOREST A, FRONT DOOR, and ROAD B. Labels reading MOTION DETECTED <> blink at the bottom of the ROAD A and ROAD B feeds, where we can see members of the Fishes removing the brush that hides the driveway to Jasper’s house.

The date overlays the upper right hand corner of the screen, 06-DEC-2027, 08:10:58.

Across the bottom is a control panel of white numbers and icons on red backgrounds.

  • A radio button control for the number of video feeds to be displayed. Though we are seeing the 4-up display, the icon does not appear to be different than the rest.
  • 16 enumerated icons, the purpose for which is unclear.
  • Video control icons for reverse, stop, play, and fast forward.
  • Three buttons with gray backgrounds and icons.
  • A wide button blinking MASTER ALARM

The scene cuts to Jasper’s rushing to the car outside the home, where none of the cacophony can be heard.

Similar to his car dashoard, it makes sense that Jasper has made this alarm himself. This might explain the clunky layout and somewhat inscrutable icons. (What do the numbers do? What about that flower on the gray background?)

The three jobs of an intruder alarm

Jasper’s alarm is OK. It certainly does the job of grabbing the household’s attention, which is the first job of an alarm, and does it without alerting the intruders, as we see in the shot outside the house.

It could do a bit better at the second job of an alarm, which is to inform the household of the nature of the problem. That they have to gather around the monitor takes precious time that could be used for making themselves safer. It could be improved by removing this requirement.

  • If Jasper had added more information to the audio alarm, even so basic as a prerecorded “Motion on the road! Motion on the road!” then they might not have needed to gather around the monitor at all.
  • If the relevant video feeds could be piped to wearable devices, phones, or their car, then they can fill in their understanding at the same time that they are taking steps to getting the hell out of there.
  • Having the artificial intelligence that we have in actual-world 2017 (much less speculative 2027), we know that narrow AI can process that video to have many more details in the broadcast message. “Motion on the road! I see three cars and at least a dozen armed men!”

There is arguably a third job of an advanced alarm, and this is to help the household understand the best course of action. This can be problematic when the confidence of the recommendation is low. But if the AI can confidently make a recommendation, it can use whatever actuators it has to help them along their way.

  • It could be informational, such as describing the best option. The audio alarm could encourage them to “Take the back road!” It could even alert the police (though in the world of Children of Men, Jasper would not trust them and they may be disinclined to care.)
  • The alarm could give some parameters and best-practice recommendations like, “You have 10 minutes to be in the car! Save only yourselves, carry nothing!”
  • It could keep updating the situation and the countdown so the household does not have to monitor it.
  • It can physically help as best it can, like remotely starting and positioning cars for them.

This can get conceptually tricky as the best course of action may be conditional, e.g. “If you can get to the car in 5 minutes, then escape is your best option, but if it takes longer or you have defenses, then securing the home and alerting the police is the better bet.” But that may be too much to process in the moment, and for a household that does not rehearse response scenarios, the simpler instruction may be safer.

Kubris

Perhaps the most unusual interface in the film is a game seen when Theo visits his cousin Nigel for a meal and to ask for a favor. Nigel’’s son Alex sits at the table silent and distant, his attention on a strange game that it’s designer, Mark Coleran, tells me is called “Kubris,” a 3D hybrid of Tetris and Rubik’s Cube.

ChildrenofMen-018.jpg

Alex operates the game by twitching and sliding his fingers in the air. With each twitch a small twang is heard. He suspends his hand a bit above the table to have room. His finger movements are tracked by thin black wires that extend from small plastic discs at his fingertips back to a device worn on his wrist. This device looks like a streamlined digital watch, but where the face of a clock would be are a set of multicolored LEDs arranged in rows.  These LEDs flicker on and off in inscrutable patterns, but clearly showing some state of the game. There is an inset LED block that also displays an increasing score.

The game also features a small, transparent, flat screen that rests on the table in front of him. It displays a computer-generated cube, similar to a 5×5 Rubik’s Cube, made up of smaller transparent cubes that share colors with the LEDs on his wrist. As Alex plays, he changes the orientation of the cube, and positions smaller cubes along the surface of the larger.

ChildrenofMen-022.jpg

Alex plays this game continually during the course of the scene. He is so engrossed in it that when Nigel asks him twice to take his pills, he doesn’t even register the instruction. Nigel must yell at him to get Alex to comply.

Though the exact workings of the game are a mystery, it serves to illustrate in a technological way how some of the younger people in 2027 disengage from the horror of the world through games that have been designed for addiction and obsession.

Bot envoys (for extremely-high-latency communications)

While recording a podcast with the guys at DecipherSciFi about the twee(n) love story The Space Between Us, we spent some time kvetching about how silly it was that many of the scenes involved Gardner, on Mars, in a real-time text chat with a girl named Tulsa, on Earth. It’s partly bothersome because throughout the rest of the the movie, the story tries for a Mohs sci-fi hardness of, like, 1.5, somewhere between Real Life and Speculative Science, so it can’t really excuse itself through the Applied Phlebotinum that, say, Star Wars might use. The rest of the film feels like it’s trying to have believable science, but during these scenes it just whistles, looks the other way, and hopes you don’t notice that the two lovebirds are breaking the laws of physics as they swap flirt emoji.

Hopefully unnecessary science brief: Mars and Earth are far away from each other. Even if the communications transmissions are sent at light speed between them, it takes much longer than the 1 second of response time required to feel “instant.” How much longer? It depends. The planets orbit the sun at different speeds, so aren’t a constant distance apart. At their closest, it takes light 3 minutes to travel between Mars and Earth, and at their farthest—while not being blocked by the sun—it takes about 21 minutes. A round-trip is double that. So nothing akin to real-time chat is going to happen.

But I’m a designer, a sci-fi apologist, and a fairly talented backworlder. I want to make it work. And perhaps because of my recent dive into narrow AI, I began to realize that, well, in a way, maybe it could. It just requires rethinking what’s happening in the chat.

Let’s first acknowledge that we’ve solved long distance communications a long time ago. Gardner and Tulsa could just, you know, swap letters or, like the characters in 2001: A Space Odyssey, recorded video messages. There. Problem solved. It’s not real-time interaction, but it gets the job done. But kids aren’t so much into pen pals anymore, and we have to acknowledge that Gardner doesn’t want to tip his hand that he’s on Mars (it’s a grave NASA secret, for plot reasons). So the question is how could we make it work so it feels like a real time chat to her. Let’s first solve it for the case where he’s trying to disguise his location, and then how it might work when both participants are in the know.

Fooling Tulsa

Since 1984 (ping me, as always, if you can think of an earlier reference) sci-fi has had the notion of a digitally-replicated personality. Here I’m thinking of Gibson’s Neuromancer and the RAM boards on which Dixie Flatline “lives.” These RAM boards house an interactive digital personality of a person, built out of a lifetime of digital traces left behind: social media, emails, photos, video clips, connections, expressed interests, etc. Anyone in that story could hook the RAM board up to a computer, and have conversations with the personality housed there that would closely approximate how that person would (or would have) respond in real life.

SBU_Tulsa.png
Listen to the podcast for a mini-rant on translucent screens, followed by apologetics.

Is this likely to actually happen? Well it kind of already is. Here in the real world, we’re seeing early, crude “me bots” populate the net which are taking baby steps toward the same thing. (See MessinaBot, https://bottr.me/, https://sensay.it/, the forthcoming http://bot.me/) By the time we actually get a colony to Mars (plus the 16 years for Gardner to mature), mebot technology should should be able to stand in for him convincingly enough in basic online conversations.

Training the bot

So in the story, he would look through cached social media feeds to find a young lady he wanted to strike up a conversation with, and then ask his bot-maker engine to look at her public social media to build a herBot with whom he could chat, to train it for conversations. During this training, the TulsaBot would chat about topics of interest gathered from her social media. He could pause the conversation to look up references or prepare convincing answers to the trickier questions TulsaBot asks. He could also add some topics to the conversation they might have in common, and questions he might want to ask her. By doing this, his GardnerBot isn’t just some generic thing he sends out to troll any young woman with. It’s a more genuine, interactive first “letter” sent directly to her. He sends this GardnerBot to servers on Earth.

Hey-mars-chat.gif
A demonstration of a chat with a short Martian delay. (Yes, it’s an animated gif.)

Launching the bot

GardnerBot would wait until it saw Tulsa online and strike up the conversation with her. It would send a signal back to Gardner that the chat has begun so he can sit on his end and read a space-delayed transcript of the chat. GardnerBot would try its best to manage the chat based on what it knows about awkward teen conversation, Turing test best practices, what it knows about Gardner, and how it has been trained specifically for Tulsa. Gardner would assuage some of his guilt by having it dodge and carefully frame the truth, but not outright lie.

Buying time

If during the conversation she raised a topic or asked a question for which GardnerBot was not trained, it could promise an answer later, and then deflect, knowing that it should pad the conversation in the meantime:

  • Ask her to answer the same question first, probing into details to understand rationale and buy more time
  • Dive down into a related subtopic in which the bot has confidence, and which promises to answer the initial question
  • Deflect conversation to another topic in which it has a high degree of confidence and lots of detail to share
  • Text a story that Gardner likes to tell that is known to take about as long as the current round-trip signal

Example

  • TULSA
  • OK, here’s one: If you had to live anywhere on Earth where they don’t speak English, where would you live?

GardnerBot has a low confidence that it knows Gardner’s answer. It could respond…

  1. (you first) “Oh wow. That is a tough one. Can I have a couple of minutes to think about it? I promise I’ll answer, but you tell me yours first.”
  2. (related subtopic) “I’m thinking about this foreign movie that I saw one time. There were a lot of animals in it and a waterfall. Does that sound familiar?”
  3. (new topic) “What? How am I supposed to answer that one? 🙂 Umm…While I think about it, tell me…what kind of animal would you want to be reincarnated as. And you have to say why.”
  4. (story delay) “Ha. Sure, but can I tell a story first? When I was a little kid, I used to be obsessed with this music that I would hear drifting into my room from somewhere around my house…”

Lagged-realtime training

Each of those responses is a delay tactic that allows the chat transcript to travel to Mars for Gardner to do some bot training on the topic. He would be watching the time-delayed transcript of the chat, keeping an eye on an adjacent track of data containing the meta information about what the bot is doing, conversationally speaking. When he saw it hit low-confidence or high-stakes topic and deflect, it would provide a chat window for him to tell the GardnerBot what it should do or say.

  • To the stalling GARDNERBOT…
  • GARDNER
  • For now, I’m going to pick India, because it’s warm and I bet I would really like the spicy food and the rain. Whatever that colored powder festival is called. I’m also interested in their culture, Bollywood, and Hinduism.
  • As he types, the message travels back to Earth where GardnerBot begins to incorporate his answers to the chat…
SBU_Gardner.png
  • At a natural break in the conversation…
  • GARDNERBOT
  • OK. I think I finally have an answer to your earlier question. How about…India?
  • TULSA
  • India?
  • GARDNERBOT
  • Think about it! Running around in warm rain. Or trying some of the street food under an umbrella. Have you seen youTube videos from that festival with the colored powder everywhere? It looks so cool. Do you know what it’s called?

Note that the bot could easily look it up and replace “that festival with the colored powder everywhere” with “Holi Festival of Color” but it shouldn’t. Gardner doesn’t know that fact, so the bot shouldn’t pretend it knows it. A Cyrano-de-Bergerac software—where it makes him sound more eloquent, intelligent, or charming than he really is to woo her—would be a worse kind of deception. Gardner wants to hide where he is, not who he is.

That said, Gardner should be able to direct the bot, to change its tactics. “OMG. GardnerBot! You’re getting too personal! Back off!” It might not be enough to cover a flub made 42 minutes ago, but of course the bot should know how to apologize on Gardner’s behalf and ask conversational forgiveness.

Gotta go

If the signal to Mars got interrupted or the bot got into too much trouble with pressure to talk about low confidence or high stakes topics, it could use a believable, pre-rolled excuse to end the conversation.

  • GARDNERBOT
  • Oh crap. Will you be online later? I’ve got chores I have to do.

Then, Gardner could chat with TulsaBot on his end without time pressure to refine GardnerBot per their most recent topics, which would be sent back to Earth servers to be ready for the next chat.

In this way he could have “chats” with Tulsa that are run by a bot but quite custom to the two of them. It’s really Gardner’s questions, topics, jokes, and interest, but a bot-managed delivery of these things.

So it could work, does it fit the movie? I think so. It would be believable because he’s a nerd raised by scientists. He made his own robot, why not his own bot?

From the audience’s perspective, it might look like they’re chatting in real time, but subtle cues on Gardner’s interface reward the diligent with hints that he’s watching a time delay. Maybe the chat we see in the film is even just cleverly edited to remove the bots.

How he manages to hide this data stream from NASA to avoid detection is another question better handled by someone else.

SBU_whodis.png

An honest version: bot envoy

So that solves the logic from the movie’s perspective but of course it’s still squickish. He is ultimately deceiving her. Once he returns to Mars and she is back on Earth, could they still use the same system, but with full knowledge of its botness? Would real world astronauts use it?

Would it be too fake?

I don’t think it would be too fake. Sure, the bot is not the real person, but neither are the pictures, videos, and letters we fondly keep with us as we travel far from home. We know they’re just simulacra, souvenir likenesses of someone we love. We don’t throw these away in disgust for being fakes. They are precious because they are reminders of the real thing. So would the themBot.

  • GARDNER
  • Hey, TulsaBot. Remember when we were knee deep in the Pacific Ocean? I was thinking about that today.
  • TULSABOT
  • I do. It’s weird how it messes with your sense of balance, right? Did you end up dreaming about it later? I sometimes do after being in waves a long time.
  • GARDNER
  • I can’t remember, but someday I hope to come back to Earth and feel it again. OK. I have to go, but let me know how training is going. Have you been on the G machine yet?

Nicely, you wouldn’t need stall tactics in the honest version. Or maybe it uses them, but can be called out.

  • TULSA
  • GardnerBot, you don’t have to stall. Just tell Gardner to watch Mission to Mars and update you. Because it’s hilarious and we have to go check out the face when I’m there.

Sending your loved one the transcript will turn it into a kind of love letter. The transcript could even be appended with a letter that jokes about the bot. The example above was too short for any semi-realtime insertions in the text, but maybe that would encourage longer chats. Then the bot serves as charming filler, covering the delays between real contact.

Ultimately, yes, I think we can backworld what looks physics-breaking into something that makes sense, and might even be a new kind of interactive memento between interplanetary sweethearts, family, and friends.

Sci-fi Interfaces and Decipher SciFi go to the movies

OMG y’all. We totally got asked on a date and we should totally go.

So I happen to be in NYC for the Interaction17 conference this week, and agreed with the guys from the Decipher SciFi podcast that we should hang out. So it’s late notice, but we have a plan: Join us at 7:25 P.M. to watch The Space Between Us, and then hangout and chat about it afterward? There may even be podcast recording and interface redesigning, it’s hard to say. Providing you’re not into The Big Game.

Here’s a link to the event details.

https://www.facebook.com/events/1302970839741111/1304373202934208/?notif_t=like&notif_id=1486263480732356

TSBU.jpg

Luke’s predictive HUD

When Luke is driving Kee and Theo to a boat on the coast, the car’s heads-up-display shows him the car’s speed with a translucent red number and speed gauge. There are also two broken, blurry gauges showing unknown information.

Suddenly the road becomes blocked by a flaming car rolled onto the road by a then unknown gang. In response, an IMPACT warning triangle zooms in several times to warn the driver of the danger, accompanied by a persistent dinging sound.

childrenofmen-impact-08

It commands attention effectively

Props to this attention-commanding signal. Neuroscience tells us that symmetrical expansion like this triggers something called a startle response.  (I first learned this in the awesome and highly recommended book Mind Hacks.) Any time we see symmetrical expansion in our field of vision, within milliseconds our sympathetic nervous system takes over, fixes our attention to that spot, and prompts us to avoid the thing that our brains believe is coming right at us. It all happens way before conscious processing, and that’s a good thing. It’s evolutionarily designed to keep us safe from falling rocks, flying fists, and pouncing tigers, and scenarios like that don’t have time for the relatively slow conscious processes.

Well visualized

The startle response varies in strength depending on several things.

  • The anxiety of the person (an anxious person will react to a slighter signal)
  • The driver’s habituation to the signal
  • The strength of the signal, in this case…
    • Contrast of the shape against its background
    • The speed of the expansion
  • The presence of a prepulse stimulus

We want the signal to be strong enough to grab the attention of a possibly-distracted driver, but not strong enough to cause them to overreact and risk control of car. While anything this critical to safety needs to be thoroughly tested, the size of the IMPACT triangle seems to sit in the golden mean between these two.

And while the effect is strongest in the lab with a dark shape expanding over a light background, I suspect given habituation to the moving background of the roadscape and a comparatively static HUD, the sympathetic nervous system would have no problem processing this light-on-dark shape.

Well placed

We only see it in action once, so we don’t know if the placement is dynamic. But it appears to be positioned on the HUD such that it draws Luke’s attention directly to the point in his field of vision where the flaming car is. (It looks offset to us because the camera is positioned in the middle of the back seat rather than the driver’s seat.) This dynamic positioning is great since it saves the driver critical bits of time. If the signal was fixed, then the driver would have his attention pulled between the IMPACT triangle and the actual thing. Much better to have the display say, “LOOK HERE!”

Readers of the book will recall this nuance from the lesson from Chapter 8, Augment the Periphery of Vision: “Objects should be placed at the edge of the user’s view when they are not needed, and adjacent to the locus of attention when they are.”

Improvements

There are a few improvements that could be made.

  • It could synchronize the audio to the visual. The dinging is dissociated from the motion of the triangle, and even sounds a bit like a seat belt warning rather than something trying to warn you of a possible, life-threatening collision. Having the sound and visual in sync would strengthen the signal. It could even increase volume with the probability and severity of impact.
  • It could increase the strength of the audio signal by suppressing competing audio, by pausing any audio entertainment and even canceling ambient sounds.
  • It could predict farther into the future. The triangle only appears once the flaming car actually stops in the road a few meters ahead. But there is clearly a burning car rolling down to the road for seconds before that. We see it. The passengers see it. Better sensors and prediction models would have drawn Luke’s attention to the problem earlier and helped him react sooner.
  • It could also know when the driver is actually focused on the problem and than fade the signal to the periphery so that it does not cover up any vital visual information. It can then fade completely when the risk has passed.
  • An even smarter system might be able to adjust the strength of the signal based on real-time variables, like the anxiety of the driver, his or her current level of distraction, ambient noise and light, and of course the degree of risk (a tumbleweed vs. a small child on the road).
  • It could of course go full agentive and apply the brakes or swerve if the driver fails to take appropriate action in time.

Despite these improvements, I believe Luke’s HUD to be well designed that gets underplayed in the drama and disorientation of the scene.

childrenofmen-impact-09