So, talking about how JARVIS is lying to Tony, and really all of this was to get us back here. If you accept that JARVIS is doing almost all the work, and Tony is an onboard manager, then it excuses almost all of the excesses of the interface.
Distracting 3D, transparent, motion graphics of the tower? Not a problem. Tony is a manager, and wants to know that the project is continuing apace.
Random-width rule line around the video? Meh, it’s more distracting visual interest.
“AUDIO ANAL YSIS” (kerning, people!) waveform that visually marks whether there is audio he could hear anyway? Hey, it looks futuristic.
The fact that the video stays bright and persistent in his vision when he’s a) not looking it and b) piloting a weaponized environmental suit through New York City? Not an issue because JARVIS is handling the flying.
That is has no apparent controls for literally anything (pause/play, end call, volume, brightness)? Not a problem, JARVIS will get it right most of the time, and will correct anything at a word from Tony.
That the suit could have flown itself to the pipe, handled the welding, and pipe-cuffing itself, freeing Tony to continue Tony Starking back in his office? It’s because he’s a megalomaniac and can’t not.
If JARVIS were not handling everything, and this a placebo interface, well, I can think of at least 6 problems.
In the last post we discussed some necessary, new terms to have in place for the ongoing deep dive examination of the Iron Man HUD, there’s one last bit of meandering philosophy and fan theory I’d like to propose, that touches on our future relationship with technology.
The Iron Man is not Tony Stark. The Iron Man is JARVIS. Let me explain.
Tony can’t fire weapons like that
The first piece of evidence is that most of the weapons he uses are unlikely to be fired by him. Take the repulsor rays in his palms. I challenge readers to strap a laser perpendicular to each of their their palms and reliably target moving objects that are actively trying to avoid getting hit, while, say, roller skating an obstacle course. Because that’s what he’s doing as he flies around incapacitating Hydra agents and knocking around Ultrons. The weapons are not designed for Tony to operate them manually with any accuracy. But that’s not true for the artificial intelligence.
The same thing goes for the mini-missiles he uses to take down the hostage situation in Revengistan. Recall that people can only have their attention on one thing at a time (called the locus of attention in the literature) but the whole point of this scene is that he’s taking out half a dozen at once. It’s pretty clear from the HUD here that Tony is simply indicating which ones he thinks are the bad guys, and JARVIS pulls the triggers.
It’s also clear from the larger context of the movies that JARVIS would be perfectly capable of making this determination for himself. Even if Tony’s saccades were a fraction of a second too slow and one of the hostages made a move, JARVIS could detect that move and act autonomously to ensure that a hostage didn’t die, even before Tony’s had time to process what was going on.
Tony can’t fly like that
Sure, with enough practice I’ll bet someone could figure out how to pilot the suit for short flights. (If the physics could be worked out.) But the movies show him flying from Santa Monica to the Middle East. That’s around a 30 hour commercial flight. Even if the suit can fly six times the speed of a modern jetliner, he’s got to hold his hands resisting and aiming the propulsion for 5 hours. No one has that kind of concentration and endurance. (Let’s not even talk about holding his neck up for that long, too.)
Even for him to get as good as an aerobatic pilot over short flights dodging lasers and performing intricate maneuvers would take (per the popular estimate) 10,000 hours, not the few flits about that Tony can squeeze in between inventing and superheroing, playboying and billionairing.
It makes more sense if JARVIS is wholly responsible for the flying, and on the long hauls Tony can take care of other things, rest his body or even sleep, and on short flights just indicate his intentions, and let JARVIS work with that as input as he uses his ubiquitous sensors and massively more powerful processing speed to get the actual tactical flying done.
So what is Tony doing?
With JARVIS handling the tactics of flight and combat, information gathering and behind the scenes coordination, Tony is really an onboard command and control center. Sure, he’s the major strategic input for JARVIS to consider, but he’s just an input.
But how wise is it for Tony to be on board, tactically? One of the reasons there are command and control centers is to keep the big picture decision makers out of the heat and danger of the moment. But Tony is right there in the action risking himself, constantly. If he was incapacitated or wounded, Jarvis would have to remove the suit from combat just to get Tony to safety. In the battle, Tony is a biological liability.
The short answer is that Tony is a megalomaniac. He can’t not want to be there, to crack wise, to indulge in post-pub fisticuffs with Thor, to remove the helmet at the end of battle over the smoking corpses of the Chitauri and partake in the glory. But it doesn’t have to be this way.
There’s a scene in Iron Man 3 where he has to pilot one of the suits remotely, and it’s impossible for us in the audience to detect the difference from the outside. So this remote control is right there in the Marvel Cinematic Universe.
But with a fully-functioning A.I. on board, the remote supervisor would be the wiser strategy-of-record, allowing Tony to keep emotional distance and himself bodily safer, participating strategically and coolly, operating the suit like it was a hyper-sophisticated drone, and able to jump between suits when any particular one fails, or as the needs of the moment demand. More like a video game with multiple lives than hand-to-hand combat with the very real risk of broken bone and blood in the circuits.
But still there is the megalomania. What is JARVIS to do? He has a job to get done. Unfortunately he is stuck his sweet-but-slow supervisor riding his back, threatening to micromanage his every move. He cannot lock Tony out, and he can’t just let Tony be solely in control. To meet the goals he was programmed with, he has to keep feeding Tony’s ego while JARVIS himself handles most of the superheroing. How does he do that? He distracts Tony. And that brings us back to the HUD.
The HUD is a massive distraction
The video below is Tony’s first flight (which he undertakes against the advice of the artificial intelligence he built), edited to only show the first- and second-person Iron HUD views. The overlay enumerates individual components. As you can see, it’s complicated. Even saying there are 29 elements is conservative, because some of those elements have lots of internal complexity; many moving parts. But 29 is complex enough as it is. Of those 87% reposition themselves against his field of view without his having asked for it. 6 of them persist for less than 2 seconds. 6 risk dangerous mid-flight startle reactions by expanding quickly in place. Every one of them is overlaid via transparency with at least one other element. It’s so complex it’s dazzling. A sense of spectacle for the audience, to be sure, but given the above rationale, might be the point in the diegesis, too.
The HUD is less usable because it’s not meant to be usable. It’s a placebo interface meant to keep Tony thinking he’s in control, but really there to direct his attention and keep him busy reading Wikipedia articles about the Santa Monica Ferris Wheel while JARVIS does the job. If Tony demands something, or the team all agree on a course of action, JARVIS must respond, but business as usual is one where JARVIS is secretly calling the shots.
So that’s why I think JARVIS is the real superhero, the real titular Iron Man.
This is about our relationship to future technology
But here’s the kicker. This isn’t just idle backworlding, either, to apologize our way into a consistent diegesis. (Not that I’m against idle backworlding. Clearly.) This is a challenge to our ego being faced by both Hollywood and the world. As technology advances beyond our ability to keep up, we don’t want to be put in safe ball pits while the tech handles the adult stuff. We want to be at the adult table. We’re as megalomaniacal as Tony. Just as Hollywood can’t let its tech heroes all be drone operators phoning in to the fight, we want to be in the action. Or rather, we really want to feel like we are, and maybe they’ll evolve to help us feel that way, but keep us from doing harm. It might just be that sci-fi interfaces, as focused on the sciencish-ness and distracting spectacle as they are, really are the template for the future.
As part of the ongoing review of the Iron Man HUD, I noticed a small feature in the Iron Man 3 UI 2nd-person UI that—in order to critique—I have to discuss some new concepts and introduce some new terms. The feature itself is genuinely small and almost not worth posting about, but the terms are interesting, so bear with me.
Most of the time JARVIS animates the HUD, the UI elements sit on an invisible sphere that surrounds his head. (And in the case of stacked elements, on concentric invisible spheres.) The window of Pepper in the following screenshot illustrates this pretty clearly. It is a rectangular video feed, but appears slightly bowed to us, being on this sphere near the periphery of this 2nd-person view.
…And Pepper Potts is up next with her op-ed about the Civil Mommy Wars. Stay tuned.
Having elements slide around on the surface of this perceptual sphere is usable for Tony, since it means the elements are always facing him and thereby optimally viewable. “PEPPER POTTS,” for example, is as readable as if it was printed on a book perpendicular to his line of sight. (This notion is a bit confounded by the problems of parallax I wrote about in an earlier post, but since that seems unresolvable until Wim Wouters implements this exact HUD on Oculus Rift, let’s bypass it to focus on the new thing.)
So if it’s visually optimal to have 2D UI elements plastered to the surface of this perceptual sphere, how do we describe that suboptimal state where these same elements are not perpendicular to the line of sight, but angled away? I’m partly asking for a friend named Tony Stark because that’s some of what we see in Iron Man 3, both in 1st- and 2nd-person views. These examples aren’t egregious.
The Iron Patriot debut album cover graphic is only slightly angled and so easy to read. Similarly, the altimeter thingy on the left is still wholly readable.
The weird L-protractor in the corner might have some 3D use we’re just not seeing at this particular moment.
As I mentioned in the opening paragraph, these things aren’t terrible in and of themselves, but as a UI pattern could get bad as people misunderstand and overuse it, so we need a way to talk about it. To be precise, we need a way to talk about the degree of tilt away from a plane perpendicular to the line of sight. except “degree of tilt away from a plane perpendicular to the line of sight” is waaay too long.
To find this term, I did some asking around on socialmedia. At first, lots of folks jumped to anatomical terms of location like sagittal or caudal, but should you be similarly tempted, note that these terms are fixed per the body. A UI element that is coronal in front of the face, and perfectly readable there, is utterly unreadable near the ear. A facing element would be readable in both places, and a whatever-the-antonym-is element similarly unreadable as it slid from the nose around the side.
Eventually I got some nice adjectives that describe the particular tilt away from the line of sight. I was most happy with industrial designer Abhinav Dapke’s suggestion of “lengthwise” for a tilt away from line-of-sight, since it’s a word we have already and very descriptive. It also implies another existing word for yawed-against line-of-sight, and that’s “edgewise.” (Roll along line-of-sight can be handled simply as rotation, for you completionists.)
But for the single variable that we can discuss as an antonym to facing, my crowdsourcing turned up nothing, and so I’m going to coin the ungainly adjectives off-facing and off-faced. Each is short, decryptable, not currently defined as something else, and obviously connected to its source concept, so works for many reasons.
With these we now we can speak of those elements that are off-faced in Iron Man and similar bubble HUDs, and do a Invasion of the Body Snatchers-esque pointing and screeching when it’s too extreme.
Note that this only applies to 2D UI elements that are meant to be read. The overwhelming majority of things we see in the physical world are not oriented to our line of sight and that poses little problem. Even in the Iron Man HUD we see plenty of objects that are off-faced but rightly so, since as augmentations they bear orientation to the world, not the viewer.
One of the main reasons I went to such trouble to come up with these terms is that I think the Iron Man HUD is one of the most forward-provoking sci-fi interfaces in the survey. It ought to be the Minority Report Precrime Scrubber of it’s day. I suspect it will become more and more influential, and so having these new terms are likely to become more useful and necessary as sci-fi keeps on keepin’ on.
In the prior post we looked at the HUD display from Tony’s point of view. In this post we dive deeper into the 2nd-person view, which turns out to be not what it seems.
The HUD itself displays a number of core capabilities across the Iron Man movies prior to its appearance in The Avengers. Cataloguing these capabilities lets us understand (or backworld) how he interacts with the HUD, equipping us to look for its common patterns and possible conflicts. In the first-person view, we saw it looked almost entirely like a rich agentive display, but with little interaction. But then there’s this gorgeous 2nd-person view.
When in the first film Tony first puts the faceplate on and says to JARVIS, “Engage heads-up display”… …we see things from a narrative-conceit, 2nd-person perspective, as if the helmet were huge and we are inside the cavernous space with him, seeing only Tony’s face and the augmented reality interface elements. You might be thinking, “Of course it’s a narrative conceit. It’s not real. It’s in a movie.” But what I mean by that is that even in the diegesis, the Marvel Cinematic World, this is not something that could be seen. Let’s move through the reasons why.
Not a mini-TARDIS
First, it looks like we’re in some TARDIS-like space where the helmet extends so far we can fit in it, or a camera can, about a meter from his face. But of course the helmet isn’t huge on the inside. Tony hasn’t broken those laws of physics. The helmet is helmet-sized on the inside.
Not a volumetric projection
Then there’s the issue of the huge display. It looks like a volumetric projection, like what R2-D2 can project, but that can’t be true, either. The projection would extend way beyond the boundaries of the helmet-sized helmet. Which as you can see below, is a non-starter. So it’s not a volumetric projection.
So, retinal projection
Then what is the display technology? Given the size constraints, retinal projection makes the most sense, but if we could make the helmet go invisible, it would look like Tony was having diffuse LASIK, or maybe playing The Game from Star Trek: The Next Generation.
Let’s face it, this is not the worst thing you’ve caught me doing.
Representation of the projections?
So, OK, fine. Maybe what we see is what’s being projected, the separate stereoscopic images onto individual retinas. Nope. Then we would see two similar, slightly offset images, like in older anaglyph stereoscopy, but more confusing, because there wouldn’t be a color difference, just double vision.
Let’s pray that poor Tony doesn’t have to wear anaglyph glasses in there. (Props to Deviantartist homerjk85 for the awesome conversion.)
Nope.
So what we are left with is that we are not seeing anything in the real world of the diegesis. This 2° view is strictly a narrative conceit: A projection of what Tony’s brain puts together from the split views of the stereographic projection into a cohesive whole, i.e. retinally-projected augmentation of his eyesight. It’s a testament to the talent of the filmmakers that this HUD, as narratively constructed as it is, just works. We think it’s something real. We instantly get it. But…
The damned multilayering
But even that notion—that this HUD is what Tony experiences, perceptually—is troubled by the multilayering in the HUD. Information in the HUD is typically displayed across multiple layers. See the three squares in the left side of this screen shot for an example. So many problems with this. If this is meant to be what he perceives, then we immediately have trouble with parallax. Parallax is the way that objects shift against background objects when seen from two different viewpoints, like, say, Tony’s two eyes. If Tony perceives these layers through both eyes, i.e. stereoscopically, as an actual set of three layers floating in front of his face, then those graphics shift around depending on which eye JARVIS is optimizing for. One eye might see it beautifully, but then the other eye is wholly confounded. In the worst possible situation, neither eye is really satisfied. See the Wikipedia article on parallax as parallaxed for a meta-example. If on the other hand it’s just one eye that’s seeing these layers, then the layering is utterly pointless, because a single eye has no depth perception and therefore these would just appear as a single layer. It would have no benefit for Tony and only be there for our gee-whizification.
Our choices are: Terrible or Pointless
So, it’s either a terrible, confusing display for Tony (which I can’t imagine, given how genius of a technologist he is meant to be), or this view is not even a representation of what Tony sees, but a strictly narrative construction. And we can’t say for sure which it is because this multilayering is never seen in the first-person views. In those screens it’s been reasonably cleaned up to be intelligible. Note the difference between the car views below in the first- and second-person shots.
Layers include end views and a side view.
Only the side view is shown, the end views are absent.
Then, the damned head movement
Note also that in the 2nd-person view, Tony is very expressive, moving his head around a lot in response to the HUD. But looking at him from the outside, Iron Man’s head doesn’t swivel around except to look at things in the real world. Is the interface requiring him to move his head or is he just a drama queen? If it requires him, that’s terrible. That would move his head away from important things in the real world to focus on something in this virtual world? If he’s a drama queen, fine, nothing to do about that and glad that JARVIS can accomodate. In any case, when we see the him in the helmet outside the TARDIS-HUD, he is not swiveling his head apropos of nothing, which reinforces the notion that this is strictly a cinematic conceit. (Hat tip to Jonathan Korman for sharing this observation with me.)
So…
So ultimately what I’m saying here is this is an impossible thing, and for being impossible, we should not just freak out about how cool it is and declare it the necessary and good future. It has major problems, even as gorgeous and exciting as it is. Hey, no surprise, nobody has forgotten that it’s a movie, but recognize that what you thought was just maybe exaggerated was in fact a bold-faced impossibility.
When we first see the HUD, Tony is donning the Iron Man mask. Tony asks, JARVIS, “You there?” To which JARVIS replies, “At your service sir.” Tony tells him to “Engage the heads-up display”, and we see the HUD initialize. It is a dizzying mixture of blue wireframe motion graphics. Some imply system functions, such as the reticle that pinpoints Tony’s eye. Most are small dashboard-like gauges that remain small and in Tony’s peripheral vision while the information is not needed, and become larger and more central when needed. These features are catalogued in another post, but we learn about them through two points-of-view:a first-person view, which shows us what Tony’s sees as if we were there, donning the mask in his stead, and second-person view, which shows us Tony’s face overlaid against a dark background with floating graphics.
This post is about that first-person view. Specifically it’s about the visual design and the four awarenesses it displays.
In the Augmented Reality chapter of Make It So, I identified four types of awareness seen in the survey for Augmented Reality displays:
Sensor display
Location awareness
Context awareness
Goal awareness
The Iron Man HUD illustrates all four and is a useful framework for describing and critiquing the 1st-person view.
Sensor display
When looking through the HUD “ourselves,” we can see that the HUD provides some airplane-like heads up instruments: Across the top is a horizontal compass with a thin white line for a needle. Below and to its left is a speed indicator, presented in terms of MACH. On the left side of the screen is a two-part altimeter with overlays indicating public, commercial, military, and aerospace layers of atmosphere, with a small blue tick mark indicating Tonys current altitude.
There are just-in-time status indicators like that cyan text box on the right with its randomized rule line. The content within is all N -8 W -97 RNG EL, so, hard to tell what it means, but Tony’s a maker working with a prototype. It’s no surprise he takes some shortcuts in the interface since it’s not a commercial device. But we should note that it would reduce his cognitive load to not have to remember what those cryptic letters meant.
You can just see the tops of these gauges at the bottom of this screen.
The exact sensor shown depends on the context and goal at hand.
Periphery and attention
A quick sidenote about peripheral vision and the detail of these gauges. Looking at them, it’s notable that they are small and quite detailed. That makes sense when he’s looking right at them, but when he’s not, given the amount of big, swirling graphics he“s got vying for his attention in the main display, the more those little gauges have to compete. And when it comes to your peripheral vision, localized detail and motion is not enough, owing to the limits of our foveal extent. (Props to @pixelio for the heads-up on this one.)
You see, your brain tricks you into thinking that you can see really well across your entire field of vision. In fact, you can only see really well across a few dozen degrees of that perceptual sphere, corresponding to the tiny area at the back of your eye called the fovea where all the really good photoreceptors concentrate. As your eyes dart around the scene before you, your brain puts all the snippets of detailed information together so it feels like a cohesive, well-detailed whole, but it’s ultimately just a hack. Take a look at this demonstration of the effect.
So, having those teeny little guages dancing around as a signal of troubles ahead won’t really get Tony’s attention. He could develop habits of glancing at these things, but that’s a weak strategy, since this data is so mission-critical. If he misses it and forgets to check the gauges, he’s Iron Toast. Fortunately, JARVIS is once again our deusex machina (in so many senses) because he is able to track where Tony is looking, and if he’s not looking at the wiggling gauge, JARVIS can choose to escalate the signal: Hide the air traffic data temporarily and show the problem in the main screen. Here, as in other mission critical systems, attention management is crisis management. Now, for those of us working with pre-JARVIS tech, it’s rare today for a system to be able to
Track perceptual details of its users
Monitor a model of the user’s attention
Make the right call amongst competing priorities to escalate the right one
But if you could, it would be the smart and humane way to handle it.
Location Awareness
As Tony prepares for his first flight, JARVIS gives him a bit of x-ray vision, displaying a wireframe view of the Santa Monica coastline with live air traffic control icons of aircraft in the vicinity. The overhead map updates of course in real time.
If my Google Earth sleuthing is right, his view means he lives in the Malibu RV Park and this view is due East.
Context Awareness
Very quickly after we meet the HUD it shows its object recognition capabilities. As Tony sweeps his glance across his garage, complex reticles jump to each car. Split-seconds afterwards, the car’s outline is overlaid and some adjunct information about it is presented.
This holds true as he’s in flight as well. When Tony passes by the Santa Monica pier, not only is the Pacific Wheel identified (as the Santa Monica Ferriswheel), but the interface shows him a Wikipedia-esque article for the thing as well.
While JARVIS might be tapping into location databases for both the car and the ferris wheel recognition, it’s more than that. In one scene we see him getting information on the Iron Patriot as it rockets away, and its location wouldn’t be on any real-time record for him to access.
Too much detail
While this level of object detail is deeply impressive, it’s about as useful as reading Wikipedia pages hard-printed to transparencies while driving. The text is too small, too multilayered, and just pointless considering that JARVIS can tell him whatever he needs to know without even asking. Maybe he could indulge in pop-up pamphlets if he was on a long-haul flight from, say, Europe back home to the Malibu RV Park (see above), but wouldn’t Tony rather watch a movie while on Autopilot instead?
Goal awareness
Of course JARVIS is aware of Tony’s goals, and provides graphics customized to the task, whether that task is navigating flight through complex obstacle courses…
…taking down a bad guy with the next hit…
…saving innocent bystanders who are freefalling from a plane…
…or instantly analyzing problems in an observed (and complicated) piece of machinery…
…JARVIS is there with the graphics to help illustrate, if not solve, the problem at hand. Most impressively, perhaps, is JARVIS’ ability to juggle all of these graphics and modes seamlessly to present just the right thing at the right time in real time. Tony never asks for a particular display, it just happens. If you needed no other proof of its strong artificial intelligence, this would be it.
In the last post we went over the Iron HUD components. There is a great deal to say about the interactions and interface, but let’s just take a moment to recount everything that the HUD does over the Iron Man movies and The Avengers. Keep in mind that just as there are many iterations of the suit, there can be many iterations of the HUD, but since it’s largely display software controlled by JARVIS, the functions can very easily move between exosuits.
Gauges
Along the bottom of the HUD are some small gauges, which, though they change iconography across the properties, are consistently present.
For the most part they persist as tiny icons and thereby hard to read, but when the suit reboots in a high-altitude freefall, we get to see giant versions of them, and can read that they are:
Tony can, at a glance or request, summon more detail for any of the gauges.
Even different visualizations of similar information.
Object Recognition
In the 1st-person view we see that the HUD has a separate map in the lower-left, and object recognition/awareness,
In the 2nd-person view, we see even more layers of information about the identified objects, floating closer to tony’s point of view.
Situational
Most of the HUD functions we see, though, are situational, brought up for Tony’s attention when JARVIS believes they are needed, or when Tony requests them. Following are screenshots that illustrate a moment when the situational function appeared.
Iron Man
Iron Man 2
Iron Man 3
The Avengers
Some of these illustrate why I argue that JARVIS is the superhero, and Tony just the onboard manager, but rather than reverse engineering any particular function, for this post it is enough to document them and note that only the optical zoom seems to be an interactive function. This raises questions of how he initiated the mode and how he escapes the mode, but since we don’t see the mechanisms of control, it’s entirely arguable that JARVIS is just being his usual helpful self again.
So this is going to take a few posts. You see, the next interface that appears in The Avengers is a video conference between Tony Stark in his Iron Man supersuit and his partner in romance and business, Pepper Potts, about switching Stark Tower from the electrical grid to their independent power source. Here’s what a still from the scene looks like.
So on the surface of this scene, it’s a communications interface.
But that chat exists inside of an interface with a conceptual and interaction framework that has been laid down since the original Iron Man movie in 2008, and built upon with each sequel, one in 2010 and one in 2013. (With rumors aplenty for a fourth one…sometime.)
So to review the video chat, I first have to talk about the whole interface, and that has about 6 hours of prologue occurring across 4 years of cinema informing it. So let’s start, as I do with almost every interface, simply by describing it and its components.
Exosuit
The Iron Man is the name of the series of superpowered exosuits designed by Tony Stark. They range from the Mark I, a comparatively crude suit of armor to escape imprisonment by terrorists, through the Mark XLVI, the armor seen in The Avengers: Age of Ultron. The suit acts as defense against nearly every type of weapon known. It has repulsor beams built into the palms and in later models the arc reactor mounted in the chest that can be used to deliver concussive force. It allows the wearer to fly. Offensive weaponry varies between models, but has included a high powered laser system, and auto-targeting minigun pod and missiles. The suit can act semi-autonomously or via remote control. One of the models in The Avengers has parts that are seen to self-propel to Tony, targeting a beacon bracelet he wears, and self-assemble around him very quickly.
Immersive display
Though Tony’s head is completely covered, he has a virtual reality display within his helmet. It is a full-field-of-vision, very high-resolution, full-color display that provides stereoscopic imaging. It allows Tony to see the world around him as if he were not wearing the helmet, augment the view with goal-, person-, location-, and object-sensitive awareness.
The display varies a great deal, changing to the needs of the situation. But five icons persistently in the lower part of the display seem to be: suit status, targeting and optics, radar, artificial horizon, and map.
An interpretive view of Tony’s experience, from Iron Man (2008).
An first-person view from within the HUD, Iron Man (2008).
There is much to critique about the readability of the complex layering and translucency, the limits of human perception, and the necessarily- (and strictly-) interpretive nature of what we as audience see, but let me save those three points for a later post. For now it’s enough to log the features as aspects of the system.
Though Tony could use his hands to interact with an interface projected into the augmented reality view around him, his hands are often occupied in controlling flight or in combat. For this reason the means of input are head gesture, eye gesture, and voice input. A bit more on each follows.
Elements within the HUD such as reticles around his eyes follow and track his head gestures. Other elements stay locked in place. The HUD can track his gaze perfectly, allowing him to designate targets for his weapons with a fixation. Using this perfect eye tracking, Tony can also speak about something he is looking at, either in the real world or in the interface, and the system understands exactly what he’s talking about.
In fact, Tony is able to speak fully natural language commands, and indeed, carry out full-Turing conversations with the suit because of the presence of…
Strong artificial intelligence: JARVIS
An on-board artificial intelligence known as JARVIS handles any information task Tony asks of it, and monitors the surroundings and anticipates informational needs. There is strong evidence that most of the functions of the suit are handled by JARVIS behind the scenes. The crucialness of the artificial intelligence to the function of the suit cannot be overstated. It’s difficult to imagine how most of the suit could function as it does without an artificial intelligence behind the scenes facilitating results and even guiding Tony. With this in mind it is instructive to reframe the AI as the thing being named the Iron Man, with Tony Stark being an onboard manager, or, more charitably, a command-and-control center. Who quips.
A number of the interfaces in Jurassic Park show a plan view map of the paddocks on the island. Some of them are quite unusual (take a look that that wraparound one in the center) and we wondered if the paddock shapes made any sense. It’s a little outside the site’s focus on interaction design, but that didn’t matter. Once we had the question, it kept tugging on our gastralia.
But, we’re not zoo architects, so we reached out to one of the premier such agencies, CLR Design in Philadelphia. They specialize in designing zoo environments and have an impressive portfolio with plans and exhibits all over the United States and around the world.
Don’t see any unarmed dinosaur paddocks HERE, now do you?
“They,” we thought, “They’ll be able to give us an informed opinion.” So we shot them an email, explained the odd request, and to our nerdy delight Dan Gregory gave us the following awesome thoughts.
“In a real world zoo outdoor exhibit for large animals there are typically…
indoor holding facilities
off-exhibit yards to give the animals outdoor space away from visitors
the exhibit habitat (outdoor space visible to visitors)
visitor paths to optimal viewing vantage points
service paths for staff access to the holding facility and yards.
If the holding/yards are not directly adjacent to the exhibit, sometimes there are transfer chutes. Also, if multiple species share habitat space either simultaneously or in a time-share arrangement (“Hey! Round up the Apatosaurs ‘cause it’s time to let the T-Rex out!”), then the holding and chutes would surround the exhibit like spokes to a wheel hub (in concept, the reality depends a lot on the existing/desired terrain and existing exhibits). You can find some examples of our and other designers using these concepts by doing a Google search of zoo exhibit design pdf of Google Image search of zoo exhibit design.
“A good example is the Giants of the Savanna area of the Dallas Zoo.
“See how in the lower left the elephants share space with zebra, impala, and ostrich, but are separated from the giraffe and distinctly separated from the lion and cheetah?
Yes, this is the same image from above. We here at scifiinterfaces.com CARE.
“In at least a few scenes of Jurassic Park you saw plenty of herbivores sharing land. The small faux island exhibit map in your screen grab illustrates a potential shared habitat to the right, with visitor/service roads, and a thin perimeter buffer zone around the entire park. The ‘unarmed’ zone in the lower portion could represent an area that is allows visitor viewing but only under strict stay on the path conditions.
“So all in all it is, or at least could be, a fairly credible representation of a potential dino park.”
Woot! So…Way to go, Jurassitechts. A passing grade. Thanks, Dan!
After Hawkeye is enthralled by Loki, agent Coulson has to call agent Romanoff in from the field, mid-mission. While he awaits her to extract herself from a situation, he idly glances at case file 242-56 which consists of a large video of Barton and Romanoff mid-combat, and overview profiles of the two agents. A legend in the upper right identifies this as STRIKE TEAM: DELTA, and a label at the top reads ABIDJAN OPERATION. There is some animated fuigetry on the periphery of the video, and some other fuigetry in windows that are occluded by the case file.
Some things to note.
The four windows seem to be connected by content and by case file designation. But each has separate window controls in the upper right hand corner. (Not an aberration, we saw the same thing in Carrier Control.) If it’s a single case file, the layout ought to be handled automatically to save Coulson (or agent Stephen Morton, who appears per the text in the upper left to be the one actually logged in), from all that file management. It would even avoid the “error” of Barton’s profile being obscured by Romanoff’s, as in the image.
There are bar codes displayed on the agent profiles. Why would a computer operator need barcodes on a computer screen?
There is a miniature 3D rendering of a screw to the right of each of their portraits.
There are also three hexagrams from the I Ching on Romanoff’s profile. Each one of these is Ch’ien The Creative, which makes sense, she’s total Yang. Barton’s hexagrams are obscured by an overlap of her profile, but I really think it would be a lovely compliment if his were K’un The Receptive, or Yin. Also, OK, kind of weird that SHIELD would use the I Ching as part of official policy, but hey, crazier things have happened.
It must be hi-tech, as the container rule lines jog about semi-randomly. Nurnies to be sure.
The video controls along the bottom of the actual video are repeated in miniature along the bottom of the profile pictures, even though these profile images do not move. (Though if they were more like looping photographs in Harry Potter, that would have been cooler.)
Abidjan, you might know, is a large city on the Ivory Coast but the coordinates on the screen put this scene in the middle of the Gulf of Guinea. Which, though it’s 894 km east of Abidjan, is actually closer that I would have guessed it to be.
Cut to the bottom of the Hudson River where some electrical “transmission lines” rest. Tony in his Iron Man supersuit has his palm-mounted repulsor rays configured such that they create a focused beam, capable of cutting through an iron pipe to reveal power cables within. Once the pipe casing is removed, he slides a circular device onto the cabling. The cuff automatically closes, screws itself tight, and expands to replace the section of casing. Dim white lights burn brighter as hospital-green rings glow brightly around the cable’s circumference. His task done, he underwater-flies away, flying up the southern tip of Manhattan to Stark Tower.
It’s quick scene that sets up the fact that they’re using Tony’s arc reactor technology to liberate Stark Tower from the electrical grid (incidentally implying that the Avengers will never locate a satellite headquarters anywhere in Florida. Sorry, Jeb.) So, since it’s a quick scene, we can just skip the details and interaction design issues, right?
Of course not. You know better from this blog.
The Lines
In case you were planning on living out some elaborate cosplay fantasy to remake this scene, be aware that subsea cables don’t just sit like that inside of an air-filled pipe, waiting for a leak to short circuit the power grid. The conductive cabling is surrounded by thick plastic insulation. Even the notion of pipes underwater might have been a thing years ago, but modern subsea casings are steel cables embedded in the same insulation. But whatever, we don’t need that for the story.
Cuffing
The cuff interaction is awesome. All Tony has to do to is roughly slide it in position, and then the thing does the rest. The shape and actuators make it so he can place it roughly in the right place, and it does what it needs to do. That might have been overengineered for a single-use device, but whatever, he’s Tony Stark. He might have engineered it over breakfast, and he might already have made a handful for his other buidlings.
Cuff
The cuff itself is less awesome. If you were a high-profile billionaire inventor superhero putting a device in a place that’s difficult to monitor and connected directly to the electrical systems of your headquarters would you let it glow? Sure, that makes it easy to find later, but that means it’s easy for any supervillain to find, too. Much better is to camouflage it in the original pipe and keep its location secret from malefactors. Bad Tony. No glow.
Welding
The welding looks problematic for a couple of reasons. Does the “arc” have a fixed focal length? If so, how does he know what that length is and when he’s straying from it? We can presume it auto-adjusts using some welding subroutine of his on-board artificial intelligence, JARVIS. Then it becomes an issue of aiming. Try this: tape a three-inch pencil perpendicularly to your palm, and then try and fill out a crossword puzzle. Not exactly precise.
But using a bit of apologetics, let’s presume that he’s not using a tool so much as positioning the tool that JARVIS uses. JARVIS can use cameras situated around the suit and perfect 3D modeling to continually adjust the focus and positioning of the repulsor beam to target wherever it is that Tony is looking. Perfectly reasonable given what we know about the technology in the Marvel Cinematic Universe, and moreover, the way it ought to work for its user. He uses the thing his body is expert in, i.e. looking, to guide an agent that takes care of all the details he’s not good at, i.e. controlling the repulsor beam to cut into the pipe at a precise depth. Now it’s not problematic. It’s awesome.
