Dr. Strange’s augmented reality surgical assistant

We’re actually done with all of the artifacts from Doctor Strange. But there’s one last kind-of interface that’s worth talking about, and that’s when Strange assists with surgery on his own body.

After being shot with a soul-arrow by the zealot, Strange is in bad shape. He needs medical attention. He recovers his sling ring and creates a portal to the emergency room where he once worked. Stumbling with the pain, he manages to find Dr. Palmer and tell her he has a cardiac tamponade. They head to the operating theater and get Strange on the table.

DoctorStrange_AR_ER_assistant-02.png

When Strange passes out, his “spirit” is ejected from his body as an astral projection. Once he realizes what’s happened, he gathers his wits and turns to observe the procedure.

DoctorStrange_AR_ER_assistant-05.png

When Dr. Palmer approaches his body with a pericardiocentesis needle, Strange manifests so she can sense him and recommends that she aim “just a little higher.” At first she is understandably scared, but once he explains what’s happening, she gets back to business, and he acts as a virtual coach.

Continue reading

Jefferson Projection

SWHS-musicVP-01

When Imperial troopers intrude to search the house, one of the bullying officers takes interest in a device sitting on the dining table. It’s the size of a sewing machine, with a long handle along the top. It has a set of thumb toggles along the top, like old cassette tape recorder buttons.

Saun convinces the officer to sit down, stretches the thin script with a bunch of pointless fiddling of a volume slider and pantomimed delays, and at last fumbles the front of the device open. Hinged at the bottom like a drawbridge, it exposes a small black velvet display space. Understandably exasperated, the officer stands up to shout, “Will you get on with it?” Saun presses a button on the opened panel, and the searing chord of an electric guitar can be heard at once.

SWHS-musicVP-04

Inside the drawbridge-space a spot of pink light begins to glow, and mesmerized officer who, moments ago was bent on brute intimidation, but now spends the next five minutes and 23 seconds grinning dopily at the volumetric performance by Jefferson Starship.

During the performance, 6 lights link in a pattern in the upper right hand corner of the display. When the song finishes, the device goes silent. No other interactions are seen with it.

tappa

Many questions. Why is there a whole set of buttons to open the thing? Is this the only thing it can play? If not, how do you select another performance?Is it those unused buttons on the top? Why are the buttons unlabeled? Is Jefferson Starship immortal? How is it that they have only aged in the long, long time since this was recorded? Or was this volumetric recording somehow sent back in time?  Where is the button that Saun pressed to start the playback? If there was no button, and it was the entire front panel, why doesn’t it turn on and off while the officer taps (see above)? What do the little lights do other than distract? Why is the glow pink rather than Star-Wars-standard blue? Since volumetric projections are most often free-floating, why does this appear in a lunchbox? Since there already exists ubiquitous display screens, why would anyone haul this thing around? How does this officer keep his job?

Perhaps it’s best that these questions remain unanswered. For if anything were substantially different, we would risk losing this image, of the silhouette of the lead singer and his microphone. Humanity would be the poorer for it.

SWHS-musicVP-09

Avengers, assembly!

Avengers-lookatthis.png

When Coulson hands Tony a case file, it turns out to be an exciting kind of file. For carrying, it’s a large black slab. After Tony grabs it, he grabs the long edges and pulls in opposite directions. One part is a thin translucent screen that fits into an angled slot in the other part, in a laptop-like configuration, right down to a built-in keyboard.

The grip edge

The grip edge of the screen is thicker than the display, so it has a clear, physical affordance as to what part is meant to be gripped and how to pull it free from its casing, and simultaneously what end goes into the base. It’s simple and obvious. The ribbing on the grip unfortunately runs parallel to the direction of pull. It would make for a better grip and a better affordance if the grip was perpendicular to the direction of pull. Minor quibble.

I’d be worried about the ergonomics of an unadjustable display. I’d be worried about the display being easily unseated or dislodged. I’d also be worried about the strength of the join. Since there’s no give, enough force on the display might snap it clean off. But then again this is a world where “vibrium steel” exists, so material critiques may not be diegetically meaningful.

Login

Once he pulls the display from the base, the screen boops and animated amber arcs spin around the screen, signalling him to login via a rectangular panel on the right hand side of the screen. Tony puts his four fingers in the spot and drags down. A small white graphic confirms his biometrics. As a result, a WIMP display appears in grays and amber colors.

Avengers-asset-browser05

Briefing materials

One window on the left hand side shows a keypad, and he enters 1-8-5-4. The keypad disappears and a series of thumbnail images—portraits of members of the Avengers initiative—appear in its place. Pepper asks Tony, “What is all this?” Tony replies, saying, “This is, uh…” and in a quick gesture, places his ten fingertips on the screen at the portraits, and then throws his hands outward, off the display.

The portraits slide offscreen to become ceiling-height volumetric windows filled with rich media dossiers on Thor, Steve Rogers, and David Banner. There are videos, portraits, schematics, tables of data, cellular graphics, and maps. There’s a smaller display near the desktop where the “file” rests about the tesseract. (More on this bit in the next post.)

Briefing.gif

Insert standard complaint here about the eye strain that a translucent display causes, and the apology that yes, I understand it’s an effective and seemingly high-tech way to show actors and screens simultaneously. But I’d be remiss if I didn’t mention it.

The two-part login shows an understanding of multifactor authentication—a first in the survey, so props for that. Tony must provide something he “is”, i.e. his fingerprints, and something he knows, i.e. the passcode. Only then does the top secret information become available.

I have another standard grouse about the screen providing no affordances that content has an alternate view available, and that a secret gesture summons that view. I’d also ordinarily critique the displays for having nearly no visual hierarchy, i.e. no way for your eyes to begin making sense of it, and a lot of pointless-motion noise that pulls your attention in every which way.

But, this beat is about the wonder of the technology, the breadth of information SHIELD in its arsenal, and the surprise of familiar tech becoming epic, so I’m giving it a narrative pass.

Also, OK, Tony’s a universe-class hacker, so maybe he’s just knowledgeable/cocky enough to not need the affordances and turned them off. All that said, in my due diligence: Affordances still matter, people.

Holomax

In the town square, Marty is quite surprised by a volumetric advertisement for “JAWS 19

BttF_045

The crude-geometry shape of a cartoon shark emerges from a projection space above the cinema, billed as “Holomax,” looms large above him, and then bends down to envelop him in its jaws before disappearing in a scattering of triangles. As a piece of interactive advertising it works well for being activated by a common urban activity, and then delivering an intense experience that is easily identified after the fact as illusory, and promising the same but more in the full volumetric moving picture experience.

Alien VPs

In the volumetric projection chapter of Make It So, we note that sci-fi makers take pains to distinguish the virtual from the real most often with a set of visual treatments derived from the “Pepper’s Ghost” parlor trick, augmented with additional technology cues: translucency, a blue tint, glowing whites, supersaturated colors for wireframed objects, clear pixels and/or flicker, with optional projection rays.

Prometheus has four types of VPs that adhere to this style in varying degrees. Individual displays (with their interactions) are discussed in other posts. This collection of posts compares their styles. This particular post describes the alien VPs.

Prometheus-223

The two alien VPs are quite different from the human VPs in appearance and behavior. The first thing to note is that they adhere to the Pepper’s Ghost style more readily, with glowing blue-tinted whites and transparency. Beyond that they differ in precision and implied technology.

Precision VPs

The first style of alien VP appears in the bridge of the alien vessel, where projection technology can be built into the architecture. The resolution is quite precise. When the grapefruit-sized Earth gets close to the camera in one scene, it appears to have infinite resolution, even though this is some teeny tiny percentage of the whole display.

Prometheus-228

Glowing Pollen

The other alien VP tech is made up of small, blue-white voxels that float, move in space, obey some laws of physics, and provide a crude level of resolution. These appear in the caves of the alien complex where display tech is not present in the walls, and again as “security footage” in the bridge of the alien ship. Because the voxels obey some laws of physics, it’s easier to think of them as glowing bits of pollen.

Prometheus-211 Prometheus-140

Pollen behavior

These voxels appear to not be projections of light in space, but actual motes that float through the air. When David activates the “security footage” in the alien complex, a wave of this pollen appears and flows past him. It does not pass through him, but collides with him, each collided mote taking a moment to move around him and regain its roughly-correct position in the display. (How it avoids getting in his mouth is another question entirely.) The motes even produce a gust of wind that disturb David’s bleached coif.

Pollen inaccuracy

The individual lines of pollen follow smooth arcs through the air, but lines appear to be slightly off from one another.

Prometheus-215

This style is beautiful and unique, and conveys a 3D display technology that can move to places even where there’s not a projector in line of sight. The sci-fi makers of this speculative technology use this inaccuracy to distinguish it from other displays. But if a precise understanding of the shapes being described is useful to its viewers, of course it would be better if the voxels were more precisely positioned in space. That’s a minor critique. The main critique of this display is when it gets fed back into the human displays as an arbitrary style, as I’ll discuss in the next post about the human-tech, floating-pixel displays.

Human VPs

In the volumetric projection chapter of Make It So, we note that sci-fi makers take pains to distinguish the virtual from the real most often with a set of visual treatments derived from the “Pepper’s Ghost” parlor trick, augmented with additional technology cues: translucency, a blue tint, glowing whites, supersaturated colors for wireframed objects, clear pixels and/or flicker, with optional projection rays.

Prometheus has four types of VPs that adhere to this style in varying degrees. Individual displays (with their interactions) are discussed in other posts. This collection of posts compares their styles. This particular post describes the human VPs.

Prometheus-064

Blue-box displays

One type of human-technology VPs are the blue-box displays:

  • David’s language program
  • Halloway and Shaw’s mission briefing
  • The display in Shaw’s quarters

These adhere more closely to the Pepper’s Ghost style, being contained in a translucent blue cuboid with saturated surface graphics and a grid pattern on the sides.

Weyland-Yutani VP

The other type of human displays are the Weyland-Yutani VPs. These have translucency and supersaturated wireframes, but they do not have any of the other conventional Pepper’s Ghost cues. Instead they add two new visual cues to signal to the audience their virtualness: scaffolded transitions and edge embers.

When a Weyland-Yutani VP is turned on, it does not simply blink into view. It builds. First, shapes are described in space as a tessellated surface, made of yellow-green lines describing large triangles that roughly describe the forthcoming object or its extents. These triangles have a faint smoky-yellow pattern on their surface. Some of the lines have yellow clouds and bright red segments along their lengths. Additionally, a few new triangles extend to a point space where another piece of the projection is about to appear. Then the triangles disappear, replaced with a fully refined image of the 3D object. The refined image may flicker once or twice before settling into persistence. The whole scaffolding effect is staggered across time, providing an additional sense of flicker to the transition.

Prometheus-010

Motion in resolved parts of the VP begins immediately, even as other aspects of the VP are still transitioning on.

When a VP is turned off, this scaffolding happens in reverse, as elements decay into tessellated yellow wireframes before flickering out of existence.

Edge embers

A line of glowing, flickering, sliding, yellow-green points illustrates the extents of the VP area, where a continuous surface like flooring is clipped at the limits of the display. These continue across the duration of the playback.

A growing confidence in audiences

This slightly different strategy to distinguishing VPs from the real world indicates the filmmaker’s confidence that audiences are growing familiar enough with this trope that fewer cues are needed during the display. In this case the translucency and subtle edge embers are the only persistent cues, pushing the major signals of the scaffolding and surface flicker to the transitions.

If this trend continues and sci-fi makers become overconfident, it may confuse some audiences, but at the same time give the designers of the first real-world VPs more freedom with their appearance. They wouldn’t have to look like Star Wars’.

Something new: Projected Reflectance

One interesting detail is that when we see Vickers standing in the projection of Weyland’s office, she casts a slight reflection in the volumetric surface. It implies a technology capable of projecting not just luminance, but reflectivity as well. The ability to project volumetric mirrors hasn’t appeared before in the survey.

Prometheus_VP-0012

Lesson: Transition by importance

Another interesting detail is that when the introduction to the Mission briefing ends, the environment flickers out first, then the 2D background, then Weyland’s dog, then finally Weyland.

This order isn’t by position, brightness, motion, or even surface area (the dog confounds that.) It is by narrative importance: Foreground, background, tertiary character, primary character. The fact that the surrounding elements fade first keep your eyes glued onto the last motion (kind of like watching the last bit of sun at a sunset), which in this order is the most important thing in the feed, i.e. the human in view. If a staggered-element fade-out becomes a norm in the real world for video conferencing (or eventually VP conferencing), this cinematic order is worth remembering.

Bridge VP: Mapping

The main interface on the bridge is the volumetric projection display. This device takes up the center of the bridge and is the size of a long billiards table. It serves multiple purposes for the crew. Its later use is to display the real-time map of the alien complex.

Map of the alien complex

The redshirt geologist named Chance in the landing party uses some nifty tools to initiate mapping of the alien complex. The information is sent from these floating sensors back to the ship, which displays the results in real time.

The display of this information is rich with a saturated-color, color-coded, edge-opacity style, leaving outer surfaces rendered in a gossamer cyan, and internal features rendered in an edge-lit green wireframe. In the area above the VP surface, other arbitrary rectangles of data can be summoned for particular tasks, including in-air volumetric keyboards. The flat base of the bridge VP is mirrored, which given the complex 3D nature of the information, causes a bit of visual confusion. (Am I seeing two diamonds reflected or four on two levels?)

Later in the film, Janek tells Ravel to modify the display; specifically, to “strip away the dome” and “isolate that area, bring it up.” He is even to enlarge and rotate the alien spaceship when they find it. Ravel does these modifications this through a touch screen panel at his station, though he routes the results to the “table.” We don’t see the controls in use so can’t evaluate them. But being able to modify displays are one of the ways that people look for patterns and make sense of such information.

A major question about this interface is why this information is not routed back to the people who can use it the most, i.e. the landing party. Chance has to speak to Janek over their intercom and figure out his cardinal directions in one scene. I know they’re redshirts, but they’re already wearing high tech spacesuits. And in the image below we see that this diegesis has handheld volumetric projections. They couldn’t integrate one of those to a sleeve to help life-critical wayfinding?