Author Archives: Christopher Noessel

Grabby hologram

by

1

After Pepper tosses off the sexy bon mot “Work hard!” and leaves Tony to his Avengers initiative homework, Tony stands before the wall-high translucent displays projected around his room.

Amongst the videos, diagrams, metadata, and charts of the Tesseract panel, one item catches his attention. It’s the 3D depiction of the object, the tesseract itself, one of the Infinity Stones from the MCU. It is a cube rendered in a white wireframe, glowing cyan amidst the flat objects otherwise filling the display. It has an intense, cold-blue glow at its center.  Small facing circles surround the eight corners, from which thin cyan rule lines extend a couple of decimeters and connect to small, facing, inscrutable floating-point numbers and glyphs.

Avengers_PullVP-02.png

Wanting to look closer at it, he reaches up and places fingers along the edge as if it were a material object, and swipes it away from the display. It rests in his hand as if it was a real thing. He studies it for a minute and flicks his thumb forward to quickly switch the orientation 90° around the Y axis.

Then he has an Important Thought and the camera cuts to Agent Coulson and Steve Rogers flying to the helicarrier.

So regular readers of this blog (or you know, fans of blockbuster sci-fi movies in general) may have a Spidey-sense that this feels somehow familiar as an interface. Where else do we see a character grabbing an object from a volumetric projection to study it? That’s right, that seminal insult-to-scientists-and-audiences alike, Prometheus. When David encounters the Alien Astrometrics VP, he grabs the wee earth from that display to nuzzle it for a little bit. Follow the link if you want that full backstory. Or you can just look and imagine it, because the interaction is largely the same: See display, grab glowing component of the VP and manipulate it.

Prometheus-229 Two anecdotes are not yet a pattern, but I’m glad to see this particular interaction again. I’m going to call it grabby holograms (capitulating a bit on adherence to the more academic term volumetric projection.) We grow up having bodies and moving about in a 3D world, so the desire to grab and turn objects to understand them is quite natural. It does require that we stop thinking of displays as untouchable, uninterruptable movies and more like toy boxes, and it seems like more and more writers are catching on to this idea.

More graphics or more information?

Additionally,  the fact that this object is the one 3D object in its display is a nice affordance that it can be grabbed. I’m not sure whether he can pull the frame containing the JOINT DARK ENERGY MISSION video to study it on the couch, but I’m fairly certain I knew that the tesseract was grabbable before Tony reached out.

On the other hand, I do wonder what Tony could have learned by looking at the VP cube so intently. There’s no information there. It’s just a pattern on the sides. The glow doesn’t change. The little glyph sticks attached to the edges are fuigets. He might be remembering something he once saw or read, but he didn’t need to flick it like he did for any new information. Maybe he has flicked a VP tesseract in the past?

Augmented “reality”

Rather, I would have liked to have seen those glyph sticks display some useful information, perhaps acting as leaders that connected the VP to related data in the main display. One corner’s line could lead to the Zero Point Extraction chart. Another to the lovely orange waveform display. This way Tony could hold the cube and glance at its related information. These are all augmented reality additions.

Augmented VP

Or, even better, could he do some things that are possible with VPs that aren’t possible with AR. He should be able to scale it to be quite large or small. Create arbitrary sections, or plan views. Maybe fan out depictions of all objects in the SHIELD database that are similarly glowy, stone-like, or that remind him of infinity. Maybe…there’s…a…connection…there! Or better yet, have a copy of JARVIS study the data to find correlations and likely connections to consider. We’ve seen these genuine VP interactions plenty of places (including Tony’s own workshop), so they’re part of the diegesis.

Avengers_PullVP-05.pngIn any case, this simple setup works nicely, in which interaction with a cool media helps underscore the gravity of the situation, the height of the stakes. Note to selves: The imperturbable Tony Stark is perturbed. Shit is going to get real.

 

Avengers, assembly!

by

2

Avengers-lookatthis.png

When Coulson hands Tony a case file, it turns out to be an exciting kind of file. For carrying, it’s a large black slab. After Tony grabs it, he grabs the long edges and pulls in opposite directions. One part is a thin translucent screen that fits into an angled slot in the other part, in a laptop-like configuration, right down to a built-in keyboard.

The grip edge

The grip edge of the screen is thicker than the display, so it has a clear, physical affordance as to what part is meant to be gripped and how to pull it free from its casing, and simultaneously what end goes into the base. It’s simple and obvious. The ribbing on the grip unfortunately runs parallel to the direction of pull. It would make for a better grip and a better affordance if the grip was perpendicular to the direction of pull. Minor quibble.

I’d be worried about the ergonomics of an unadjustable display. I’d be worried about the display being easily unseated or dislodged. I’d also be worried about the strength of the join. Since there’s no give, enough force on the display might snap it clean off. But then again this is a world where “vibrium steel” exists, so material critiques may not be diegetically meaningful.

Login

Once he pulls the display from the base, the screen boops and animated amber arcs spin around the screen, signalling him to login via a rectangular panel on the right hand side of the screen. Tony puts his four fingers in the spot and drags down. A small white graphic confirms his biometrics. As a result, a WIMP display appears in grays and amber colors.

Avengers-asset-browser05

Briefing materials

One window on the left hand side shows a keypad, and he enters 1-8-5-4. The keypad disappears and a series of thumbnail images—portraits of members of the Avengers initiative—appear in its place. Pepper asks Tony, “What is all this?” Tony replies, saying, “This is, uh…” and in a quick gesture, places his ten fingertips on the screen at the portraits, and then throws his hands outward, off the display.

The portraits slide offscreen to become ceiling-height volumetric windows filled with rich media dossiers on Thor, Steve Rogers, and David Banner. There are videos, portraits, schematics, tables of data, cellular graphics, and maps. There’s a smaller display near the desktop where the “file” rests about the tesseract. (More on this bit in the next post.)

Briefing.gif

Insert standard complaint here about the eye strain that a translucent display causes, and the apology that yes, I understand it’s an effective and seemingly high-tech way to show actors and screens simultaneously. But I’d be remiss if I didn’t mention it.

The two-part login shows an understanding of multifactor authentication—a first in the survey, so props for that. Tony must provide something he “is”, i.e. his fingerprints, and something he knows, i.e. the passcode. Only then does the top secret information become available.

I have another standard grouse about the screen providing no affordances that content has an alternate view available, and that a secret gesture summons that view. I’d also ordinarily critique the displays for having nearly no visual hierarchy, i.e. no way for your eyes to begin making sense of it, and a lot of pointless-motion noise that pulls your attention in every which way.

But, this beat is about the wonder of the technology, the breadth of information SHIELD in its arsenal, and the surprise of familiar tech becoming epic, so I’m giving it a narrative pass.

Also, OK, Tony’s a universe-class hacker, so maybe he’s just knowledgeable/cocky enough to not need the affordances and turned them off. All that said, in my due diligence: Affordances still matter, people.

Correcting attribution for the medpod

by

2

Gorgeous + Catastrophic presentation card

Earlier this year I made two presentations called Gorgeous+Catastrophic, in which I show six sci-fi interfaces that are both beautiful to behold and that would be disastrous if implemented in the real world, all to illustrate why we should keep interfaces in sci-fi at arm’s length and evaluate them with a critical eye. It’s a fun talk to give. You should totally ask me to come present it at your local conference.

{Editor’s note: The original Vimeo talk was taken down, but here’s a version of the talk I gave later from YouTube}

But in one of the talks, when I introduced the first of the six examples—the medpod interface from Prometheus, starting around 06:35 in the video—I misattributed the whole design to Territory Studio. This was oversimplifying the team on a couple of levels, so let me make the formal correction and apology here.

Territory Studio did work on Prometheus, and even did work on the medpod: They did the VFX interfaces shown around 07:50, and were joined by teams from Fuel VFX and Compuhire. They did not do the on-set touchscreen that sits on the side of the medpod that Noomi Rapace/Elizabeth Shaw touches directly starting around 07:28. That was designed by Shaun Yue, working as an individual contractor. An additional complication is that George Simons, who was graphics supervisor on the film, is now with Territory, but was not then. And then, there’s the credits, which only list names, not companies, and not full teams.

Mea culpa. I should have known better, since I even have an interview with Shaun Yue on this blog about that movie. It’s a small competitive field, and proper credit is hard to get. It’s functionally advertising, so this mistake isn’t minor. My apologies to Shaun, George Simons, Rheea Aranha, John Hill, Paul Roberts, Daniel Burke, Mark Jordan, Eliot Eveson, and Adam Stevenson. If I give the same talk again I won’t make the same mistake.

The trickiness of attribution

It doesn’t excuse the mistake (and I’m glad I have this forum to right the wrong) but I will note how very difficult it is to get attribution of scifiinterfaces correctly.

Who is the designer?

A sci-fi interface is a pie with a lot of fingers in it. If it’s central to the plot, then the writer will have described what it does in the script. The director will have had (in this case) his opinions as well, well before shooting, and actors may have input after reading the script and during shooting. If it’s not central to the plot, it may have been handed to someone elsewhere in the hierarchy. Then there’s art directors, production designers, and editors all directly touching the end result of what we see on screen.

With interfaces becoming more and more part of sci-fi movie making, the teams are getting larger and more specialized. There may be one team whose responsibility are on-set interfaces that the actors see and touch. Another team might be handling the post-production interfaces that are built after principal shooting. One individual might do the graphics as static elements and another do the motion design. Final assets produced by designers may be cut up and remixed by editors (without consulting the original designers) to meet the narrative needs of the flow of the story, so what winds up on screen may not be what was originally designed.

Given all of these people, where is the line of who is and isn’t the designer? Or even the design team? Is it everyone? Is it just “the designer?” Who is that in this complicated case? Who gets the credit?

Highlighted for focus

Highlighted for focus

An informally defined role

Another part of the difficulty of attribution is that interface design is not as formalized a role as many of the other roles in movie production. If a person is listed as the Director of Photography, that has a specific meaning and set of responsibilities. But there isn’t a formal title that every production agrees to, like Director of Interfaces (even though, hey, Hollywood, maybe it’s time). Sometimes, as was the case with Prometheus, it’s a mixture of individuals and studios hired separately. Sometimes the designers are in-house employees of the studio. It’s hard to even review the credits for some movies and clearly say these people were the ones involved in the interface. That person’s name may represent the individual whose name stands in for a studio. Even Prometheus listed the individuals by name, even though I’ve confirmed through some direct conversations who were hired as individual contractors rather than individuals.

It is so very complicated.

So again, this is not an excuse, but an explication about why trying to get the attribution right is a fraught enterprise, and I did not due my due diligence when speaking about interfaces. It would be easiest to simply bypass attribution altogether, but I want to recognize people for the difficult and beautiful work they do.

This whole thing is especially on my mind as we head into the end of the year and I’m working with folks to create an award for the best sci-fi interface of the year. This will be important to get right, and clearly I’ve got to get better at doing it.

Report Card: Back to the Future Part II

by

1

Read all the Back to the Future Part II reviews in chronological order.

At least according by the hubbub today, c.f. #BacktotheFuture, this movie struck a deep chord in audiences with its tongue-in-cheek futurism and occasional forays into brilliance. And while much of social media has been poring over the film to see what it got right (the pointless interpretation of futurism, imho) let’s instead return to our little corner of nerdery and see how its interfaces fared.

BttF-Report-Card

Sci: C (2 of 4) How believable are the interfaces?

This movie didn’t really aim for a serious futurism. It’s a fun adventure comedy that toyed with a futuristic setting amongst its near and far pasts. It didn’t always aim at believable as much as it did entertaining. (This permeable 4th wall is why comedy interfaces are so hard to review, and why I’ve mostly avoided it.)

Close-up of a Mr. Fusion Home Energy Reactor, a fictional device from the 'Back to the Future' film series, featuring the logo and design details.
A vintage-style television screen displaying a news broadcast with a man in a suit, while a young person in a colorful cap watches in a diner setting.
A hand is placing a tray with a small pizza into a Black & Decker Hydrator appliance.

Mr. Fusion? Oh, we get it. A play on Mr. Coffee, but super high tech! Cafe 80s? Yeah, I guess that’s us(1985) in a nutshell. A hydrator? It’s like those new fangled microwaves only even more fangled! We were never meant to take these things seriously.

But some of the technologies were presented a little more in earnest, but weren’t thought through. Car controls are meant for navigating a vehicle across 2 dimensions, not the 3 of flight. (Or the 4th dimension of time for that matter, and Doc kind of screwed those up, too.)

BttF_137

On the other hand, the skyways are a believable and a familiar way to corral the potential chaos of millions of flying cars. (Though I should note somberly that where they were going, they did, actually, need roads.) Barring personal stations in our garages, fueling stations may evolve to become as agentive/robotic as this. And given ubiquitous thumbprint readers (and a weak sense of security), you imagine that cops, homes, and service providers would have taken full advantage of it.

A flying car navigating through a futuristic cityscape with illuminated signs showing destinations like Phoenix, Boston, and London.
Scene at a Texaco gas station featuring a vintage design and signage, with a person in a colorful cap looking up at the station.

Fi: A (4 of 4) How well do the interfaces inform the narrative of the story?

So if you’re going to tell of a future that is a few cool leaps ahead of what the audience currently knows, Back to the Future Part II kicked it. It presented technologies that made people want to live in that exact future. It still inspires people to try and actually make it so. Nike plans to release MAGs in 2016. People keep trying to make a hoverboard. That’s the mark of an impression made so deeply people are still yearning for it to be real, 28 years later.

BttF_026

The smattering of other technologies that helped make this future believable may not have been as directly inspirational, but still painted a picture of a Hill Valley infused with technology.

A person wearing a puffy jacket and holding a hoverboard stands beside a pond with buildings and people in the background.
A group of four characters dressed in futuristic outfits, with one prominently yelling. The setting appears to be a diner or restaurant with various decorations in the background.
A person in a colorful cap stands in front of a large holographic advertisement for 'Holomax Jaws 19', featuring a massive shark with a menacing expression.
A person with curly hair standing in front of a projected image of a snowy mountain landscape, framed by curtains in a room.
A person standing in front of a large screen displaying various digital images and numbers, with a cozy room and brick wall in the background.

Interfaces: C (2 of 4)
How well do the interfaces equip the characters to achieve their goals?

Interestingly, the plot didn’t depend on much technology beyond the DeLorean and the hoverboard. These two were incredibly crucial, but you can pretty much take out all of the rest and all you lose is the gee-whiz. That said, these two were very critical to the plot and the characters’ goals. And between them it’s a wash, with the DeLorean failing to even be believable in giving Doc Brown tools to navigate any of the 4 dimensions (see above), and the hoverboard being so awesome we’re still many attempts in to try and bring it to life.

A futuristic car hovering above a tree-lined street with parked vehicles.
A young boy holding a colorful hoverboard while two girls look on, one with a purple hair accessory and the other in a bright outfit, set in a vibrant street scene.

I should add a tiny shout out to the the ceiling hydroponics, too, even though it wasn’t critical to the movie’s plot, to the McFly’s family goals of having healthy fresh food available even in their urban setting. Gawsh. What will they think of next?

BttF_119

Final Grade B- (8 of 12), MUST-SEE

It’s with sorrow that those of us who first imprinted on this far out future must, as of today, put this in our past. Hopefully the next generation will get another film that so vividly paints a vision of the future so awesome that they’ll still want to make it happen in, say, 2043.

Pitch time: If you’ve enjoyed the review of BTTF2, there’s so many ways to say thanks.

  • Actually say thanks.
  • Follow the blog, twitter, and share the joy with friends and family.
  • Purchasing the English or Japanese versions of the related book (or hey both)
  • Dropping a credit in the tip jar that you see on every page
  • Invite me to do similar analyses live at your company or recommend me to your favorite conference. I’m this charming and insightful in person, too, I swear.

IMDB: https://www.imdb.com/title/tt0088763/Currently streaming on:

Videoconferencing

by

0

BttF_122

Marty Sr. answers a call from a shady business colleague shortly after coming home. He takes the call in the den on the large video screen there. As he approaches the screen, he sees a crop of a Renoir painting, “Dance at La Moulin de la Galette,” with a blinking legend “INCOMING CALL” along the bottom. When he answers it, the Renoir shrinks to a corner of the screen, revealing the live video feed with his correspondent. During the conversation, the Renoir disappears, and text appears near the bottom of the screen providing reminders about the speaker. This appears automatically, with no prompting from Marty Sr.

Needles, Douglas J.
Occupation: Sys Operations
Age: 47
Birthday: August 6, 1968
Address: 88 Oriole Rd, A6t
Wife: Lauren Anne
Children: Roberta, 23 Amy, 20
Food Prefence: Steak, Mex
Food Dislike: Fish, Tuna
Drinks: Scotch, Beer
Hobbies: Avid Basketball Fan
Sports: Jogging, Slamball, Tennis
Politics: None

A man in a brown suit stands in front of a large television, reacting to a scene featuring another man smiling. The background contains a brick wall and various objects like a globe and a sound system.
A scene featuring a man standing in front of a large screen displaying a character named Douglas J. Needles with the caption 'Occupation: Sys Operations'. The setting appears to be a home or office with a brick wall.
A large television screen displays a man's face with a caption showing his name, Douglas J. Needles, and age 47. A person in a suit stands nearby, looking at the screen.

This is an augmented reality teleconference, as mentioned in Chapter 8 of Make It So: Interaction Design Lessons from Science Fiction. See more information in that chapter. In short, it’s a particularly good example of one type of augmentation that is very useful for people having to interact with networks of people much larger than Dunbar’s number equips us for. Unfortunately, the information appears in a distracting scroll across the bottom, and is not particularly pertinent to the conversation, so could benefit from a bit of context awareness or static high-resolution display to be really useful.

When the conversation is done, the screen fades to black, and an animated AT&T logo appears as a female voice offers Thank you for using AT&T. Afterwards, another odd crop of a famous impressionist artwork appears on screen. (1887-88 Self-Portrait with Grey Felt Hatby Van Gogh, if you’re curious.) This must be a preference set by the McFlys.

During the conversation, Needles convinces Marty Sr. to front some money for a shady deal. To provide the money, he grabs his briefcase and presses one of several lit buttons on its surface. A thin reader raises from the surface on mechanized accordion arms. The slot glows white, and Marty inserts his ID card into the recess and yanks it to the right. Marty presses another button and the reader retracts.

A person sitting on a couch watching a television screen displaying a character named Douglas J. Needles with details about their preferences, including drinks and hobbies.
A scene depicting a living room with a projector displaying a character named Douglas J. Needles, who is smiling. The screen lists personal information, including children's names and ages. A man in a suit is sitting in a chair, holding a briefcase and looking at the screen.
A person sitting on a couch gestures towards a large television screen, which displays a smiling man with information about his food preferences. The room has a cozy, lived-in appearance with bookshelves, a globe, and casual furniture.
A person sitting in a living room, looking at a digital screen displaying a man's profile with food preferences and dislikes.
A person sitting at a table with a notepad, looking at a television displaying a man named Douglas J. Needles, detailing his drinks and hobbies.

Though pretty cool, this portable authenticator seems multimodal, the two objects are both of the same type, i.e. something he possesses, and so is only marginally more secure than unimodal systems.

BttF_131

Almost instantly, Marty is caught in the act and fired by his boss over the same video phone. To bring home the news, Fujitsu-san presses some button off screen and large letters YOU’RE FIRED appear overlaid on the screen, animating repeatedly for emphasis. Simultaneously, he sends Marty faxes, which Marty receives numerous times in the faxes in his suitcases and others which are installed all over the house, including the in the closet where Jennifer is hiding.

A person in a suit is shown from behind, reacting to a large screen displaying the text 'YOU'RE FIRED' in bold letters, with a dimly lit room in the background.
A black briefcase on a chair with a sign reading 'YOU'RE FIRED!!' next to it, and a yellow paper strip saying 'YOU'RE' in the background.
A paper sign reading 'YOU'RE FIRED!!!' is displayed on a wall-mounted message board.

I’m not sure what would lead Needles or Marty(2015) to engage in something illegal that is obviously and instantly discoverable by authorities, but that is not a matter of interaction design.

VR Goggles

by

1

BttF_118

At the dinner table, both Marty Jr. and Marlene have VR goggles. Marty wears his continuously, but Marlene is more polite and rests hers around her neck when with the family. When she receives a call, red LEDs flash the word “PHONE” on the outside of the goggles as they ring. This would be a useful signal if the volume were turned down or the volume was baffled by ambient sounds.

BttF_120

Marty Jr’’s goggles are on, and he announces to Marty Sr. that the phone is for him and that it’s Needles.

This implies a complete wireless caller ID system (which had only just been released to market in the United States the year before the movie was released) and a single number for the household that is distributed amongst multiple communications devices simultaneously, which was not available at the time (or hey, even now), so it’s quite forward looking. Additionally, it lets the whole social circle help manage communication requests, even if it sacrifices a bit of privacy.

Garden Center

by

2

In the center of the kitchen, mounted to the ceiling, is a “Garden Center.” Out of use, it retracts out of reach, but anyone in the family can say “Fruit, please” and the Garden Center drops down to allow fresh grapes to be plucked right off the vine. When done, Marty Jr. tells it to “retract” with a thump on it, and it retracts back up to its resting place near the ceiling.

BttF_119

This is wonderful. Responds to many types of inputs and keeps healthy, fresh fruit available to the family at any time.

Black & Decker Hydrator

by

1

BttF_113

Lorraine prepares the family a pizza using a hydrator. She opens a sealed foil package, branded “Pizza Hut,” and removes a tiny puck of a pizza, placing it in the center of a large pizza tray. She inserts the tray into a “hydrator” oven and closes the hinged front door. A small green light illuminates on its panel. She puts her mouth close to the device and instructs it to, “Hydrate level 4, please.” A red light illuminates as a bubbling sound is heard for a few seconds. Then a timer bell rings, and both lights extinguish. Lorraine removes a full-sized and fully-cooked pizza from the oven.

It could be improved by not having her have to remember and enter the level of hydration. There might be an argument that this helps the hydrator feel like they’re doing enough effort, like the legendary Betty Crocker egg story. While snopes tells us that the usual version of this is poppycock, but also references Ernest Dichter’s research in which yes, the first generation of homemakers using instant cake mixes felt that a preparation that was too easy was too indulgent. So, perhaps the hydrator is first generation, and later generations will be able to detect the hydration needed from the packaging.  

Hoverstuff

by

3

Hover technology is a thing in 2015(1985) and it appears many places.

Hoverboards

BttF_075

When Marty has troubles with Griff Tannan he borrows a young girl’’s hover scooter and breaks off its handlebar. He’s able to put his skateboarding skills to use on the resulting hover board.

Griff and his gang chases Marty on their own hover boards. Griff’s has a top of the line hover board labeled a “Pit Bull.” Though Marty clearly has to manually supply forward momentum to his, Griff’s has miniature swivel-mount jet engines that (seem to) respond to the way he shifts his weight on the board.

Hovertraction

BttF_097

George requires traction for a back problem, but this doesn’’t ground him. A hover device clamps his ankles in place and responds to foot motions to move him around.

Hover tech is ideal for leaning control, like what controls a Segway. That’s just what seems to be working in the hoverboard and hovertraction devices. Lean in the direction you wish to travel, just like walking. No modality, just new skills to learn.