Cyberspace: Beijing Hotel

After selecting its location from a map, Johnny is now in front of the virtual entrance to the hotel. The virtual Beijing has a new color scheme, mostly orange with some red.

jm-33-hotel-a

The “entrance” is another tetrahedral shape made from geometric blocks. It is actually another numeric keypad. Johnny taps the blocks to enter a sequence of numbers.

The tetrahedral keypad

jm-33-hotel-b

Note that there can be more than one digit within a block. I mentioned earlier that it can be difficult to “press” with precision in virtual reality due to the lack of tactile feedback. Looking closely, here the fingers of Johnny’s “hands” cast a shadow on the pyramid, making depth perception easier.

Something is wrong, and Johnny receives an electric shock.

jm-33-hotel-c

He reacts as if the shock is real, pulling his hands back and cursing.

In the 1980s and 1990s cyberpunk books such as Neuromancer and Hardwired and roleplaying games such as Cyberpunk and ShadowRun suggested that future virtual reality systems would be able to physically attack users, the dreaded “Black ICE”. While the more vigilant Internet copyright enforcers would probably be in favour, it seems unlikely that the liability lawyers at any computer manufacturer would allow a product that could electrocute users to be released, or that users would agree to put something like that on their hands. So this is most likely  just Johnny expressing the same frustration as a current day video gamer who loses a life in a first person shooter.

The last necessary step before being granted access is, for some reason, to reshape the pyramid.

jm-32-beijing-b-animated

Here the pyramid serves as a combination lock or puzzle as well as a keypad. It’s not obvious, but Johnny does make a small 3D rotating gesture on the entire pyramid before pulling and pushing blocks around. You can also see a second layer of structure underneath the moving shapes.

Is this an effective security system? Not really. Two-factor authentication systems rely both on knowingsomething, here a numeric code, and either havingsomething, such as a specific mobile phone or token generator, or beingsomeone, with a specific fingerprint. Reshaping the blocks is just a second thing the would-be user must know, and is just as vulnerable to being guessed as the numeric code. On the other hand, it might be enough to keep out simple-minded attacks that only try the first step.

The floorplan

The “interior” of the hotel site is first displayed as a flat plan view. This builds up incrementally, a transition known among VR developers since the film Tron came out as “rezzing up”. The completed plan then rotates into a 3D structure.

jm-34-inside-a-animated

We hear the voice feedback announce “General accounts selected” but don’t see how Johnny did this. A window expands out, and Johnny splits it in half to reveal some tabular data.

jm-34-inside-b-animated

The fax and phone records are displayed in a simple tabular view, which would not look out of place on any 1995 or indeed current day desktop computer spreadsheet. There’s no need to use 3D graphics for such this.

jm-34-inside-d

There are new interface elements here, overlaying the tabular data in pink. At the top we can read SEARCH > FAX CHARGES: FOUND. And on the right is a set of inscrutable numbers with headings GRID, LEVEL, MENU, and XYZ. This could be some orientation within the data, but it doesn’t make sense. In the lower-left we see a label for elevation, with data as “coordinates in sector 4.”

Below that a 9-key arrangement with arrow shapes. Perhaps this is a navigation aid for people using conventional 2D desktop interfaces rather than full virtual reality equipment, allowing them to move around by clicking the onscreen arrows or pressing the equivalent keys. If the keys are similar to those used in computer games, the up and down arrow keys move forward or backwards and the left and right keys rotate, assuming movement is predominantly in the horizontal plane. The other keys might be for banking or vertical movement.

Johnny searches for the outgoing fax. He does not use any graphical gestures for this, instead specifying the search date and time ranges by speaking. Words and operators are more precise than graphic symbols for this kind of database query, but typing on a virtual keyboard would be more awkward than speech.

When the particular table cell is found, he uses the fingertips of both hands to expand the contents, one of the standard gestures described in the Make It So book.

Not surprisingly for a Beijing hotel, the internal records are not in English. Johnny again uses a voice command to ask for translation.

jm-34-inside-e

The hotel record is just the metadata, not the actual images he’s looking for, suggesting that “fax” system is fully digital and the faxes themselves are treated like modern email messages and deleted once sent. The metadata does tell Johnny that the images were faxed to a online copyshop in Newark. Since it is network connected, Johnny can jump straight to it in cyberspace.

Iron Man HUD: 1st person view

In the prior post we catalogued the functions in the Iron HUD. Today we examine the 1st-person display.

When we first see the HUD, Tony is donning the Iron Man mask. Tony asks, JARVIS, “You there?” To which JARVIS replies, “At your service sir.” Tony tells him to “Engage the heads-up display”, and we see the HUD initialize. It is a dizzying mixture of blue wireframe motion graphics. Some imply system functions, such as the reticle that pinpoints Tony’s eye. Most are small dashboard-like gauges that remain small and in Tony’s peripheral vision while the information is not needed, and become larger and more central when needed. These features are catalogued in another post, but we learn about them through two points-of-view:a first-person view, which shows us what Tony’s sees as if we were there, donning the mask in his stead, and second-person view, which shows us Tony’s face overlaid against a dark background with floating graphics.

This post is about that first-person view. Specifically it’s about the visual design and the four awarenesses it displays.

Avengers-missile-fetching04

In the Augmented Reality chapter of Make It So, I identified four types of awareness seen in the survey for Augmented Reality displays:

  1. Sensor display
  2. Location awareness
  3. Context awareness
  4. Goal awareness

The Iron Man HUD illustrates all four and is a useful framework for describing and critiquing the 1st-person view.

Sensor display

When looking through the HUD “ourselves,” we can see that the HUD provides some airplane-like heads up instruments: Across the top is a horizontal compass with a thin white line for a needle. Below and to its left is a speed indicator, presented in terms of MACH. On the left side of the screen is a two-part altimeter with overlays indicating public, commercial, military, and aerospace layers of atmosphere, with a small blue tick mark indicating Tony’s current altitude.

There are just-in-time status indicators like that cyan text box on the right with its randomized rule line. The content within is all N -8 W -97 RNG EL, so, hard to tell what it means, but Tony’s a maker working with a prototype. It’s no surprise he takes some shortcuts in the interface since it’s not a commercial device. But we should note that it would reduce his cognitive load to not have to remember what those cryptic letters meant.

IronMan1_HUD08
You can just see the tops of these gauges at the bottom of this screen.

The exact sensor shown depends on the context and goal at hand.

Periphery and attention

A quick sidenote about peripheral vision and the detail of these gauges. Looking at them, it’s notable that they are small and quite detailed. That makes sense when he’s looking right at them, but when he’s not, given the amount of big, swirling graphics he“s got vying for his attention in the main display, the more those little gauges have to compete. And when it comes to your peripheral vision, localized detail and motion is not enough, owing to the limits of our foveal extent. (Props to @pixelio for the heads-up on this one.)

You see, your brain tricks you into thinking that you can see really well across your entire field of vision. In fact, you can only see really well across a few dozen degrees of that perceptual sphere, corresponding to the tiny area at the back of your eye called the fovea where all the really good photoreceptors concentrate. As your eyes dart around the scene before you, your brain puts all the snippets of detailed information together so it feels like a cohesive, well-detailed whole, but it’s ultimately just a hack. Take a look at this demonstration of the effect.

Screen Shot 2015-07-20 at 23.49.56
This only works if you view it live.

So, having those teeny little guages dancing around as a signal of troubles ahead won’t really get Tony’s attention. He could develop habits of glancing at these things, but that’s a weak strategy, since this data is so mission-critical. If he misses it and forgets to check the gauges, he’s Iron Toast. Fortunately, JARVIS is once again our deus ex machina (in so many senses) because he is able to track where Tony is looking, and if he’s not looking at the wiggling gauge, JARVIS can choose to escalate the signal: Hide the air traffic data temporarily and show the problem in the main screen. Here, as in other mission critical systems, attention management is crisis management. Now, for those of us working with pre-JARVIS tech, it’s rare today for a system to be able to

  • Track perceptual details of its users
  • Monitor a model of the user’s attention
  • Make the right call amongst competing priorities to escalate the right one

But if you could, it would be the smart and humane way to handle it.

Location Awareness

As Tony prepares for his first flight, JARVIS gives him a bit of x-ray vision, displaying a wireframe view of the Santa Monica coastline with live air traffic control icons of aircraft in the vicinity. The overhead map updates of course in real time.

IronMan1_HUD17
If my Google Earth sleuthing is right, his view means he lives in the Malibu RV Park and this view is due East.

Context Awareness

Very quickly after we meet the HUD it shows its object recognition capabilities. As Tony sweeps his glance across his garage, complex reticles jump to each car. Split-seconds afterwards, the car’’s outline is overlaid and some adjunct information about it is presented.

IronMan1_HUD10

This holds true as he’s in flight as well. When Tony passes by the Santa Monica pier, not only is the Pacific Wheel identified (as the Santa Monica Ferriswheel), but the interface shows him a Wikipedia-esque article for the thing as well.

IronMan1_HUD19

IronMan1_HUD21

While JARVIS might be tapping into location databases for both the car and the ferris wheel recognition, it’s more than that. In one scene we see him getting information on the Iron Patriot as it rockets away, and its location wouldn’t be on any real-time record for him to access.

Optical zoom

Too much detail

While this level of object detail is deeply impressive, it’s about as useful as reading Wikipedia pages hard-printed to transparencies while driving. The text is too small, too multilayered, and just pointless considering that JARVIS can tell him whatever he needs to know without even asking. Maybe he could indulge in pop-up pamphlets if he was on a long-haul flight from, say, Europe back home to the Malibu RV Park (see above), but wouldn’t Tony rather watch a movie while on Autopilot instead?

Goal awareness

Of course JARVIS is aware of Tony’s goals, and provides graphics customized to the task, whether that task is navigating flight through complex obstacle courses…

3D wayfinding

…taking down a bad guy with the next hit…

Suggested target points

…saving innocent bystanders who are freefalling from a plane…

Biometric analysis, target acquisition

…or instantly analyzing problems in an observed (and complicated) piece of machinery…

3D schematics of observed machinery with damage highlights

…JARVIS is there with the graphics to help illustrate, if not solve, the problem at hand. Most impressively, perhaps, is JARVIS’ ability to juggle all of these graphics and modes seamlessly to present just the right thing at the right time in real time. Tony never asks for a particular display, it just happens. If you needed no other proof of its strong artificial intelligence, this would be it.

Next up in the Iron HUD series: Compare and contrast the 2nd-person view.