Dr. Strange’s augmented reality surgical assistant

We’re actually done with all of the artifacts from Doctor Strange. But there’s one last kind-of interface that’s worth talking about, and that’s when Strange assists with surgery on his own body.

After being shot with a soul-arrow by the zealot, Strange is in bad shape. He needs medical attention. He recovers his sling ring and creates a portal to the emergency room where he once worked. Stumbling with the pain, he manages to find Dr. Palmer and tell her he has a cardiac tamponade. They head to the operating theater and get Strange on the table.

DoctorStrange_AR_ER_assistant-02.png

When Strange passes out, his “spirit” is ejected from his body as an astral projection. Once he realizes what’s happened, he gathers his wits and turns to observe the procedure.

DoctorStrange_AR_ER_assistant-05.png

When Dr. Palmer approaches his body with a pericardiocentesis needle, Strange manifests so she can sense him and recommends that she aim “just a little higher.” At first she is understandably scared, but once he explains what’s happening, she gets back to business, and he acts as a virtual coach.

Continue reading

Advertisements

The Dark Dimension mode (5 of 5)

We see a completely new mode for the Eye in the Dark Dimension. With a flourish of his right hand over his left forearm, a band of green lines begin orbiting his forearm just below his wrist. (Another orbits just below his elbow, just off-camera in the animated gif.) The band signals that Strange has set this point in time as a “save point,” like in a video game. From that point forward, when he dies, time resets and he is returned here, alive and well, though he and anyone else in the loop is aware that it happened.

Dark-Dimension-savepoint.gif

In the scene he’s confronting a hostile god-like creature on its own mystical turf, so he dies a lot.

DoctorStrange-disintegrate.png

An interesting moment happens when Strange is hopping from the blue-ringed planetoid to the one close to the giant Dormammu face. He glances down at his wrist, making sure that his savepoint was set. It’s a nice tell, letting us know that Strange is a nervous about facing the giant, Galactus-sized primordial evil that is Dormammu. This nervousness ties right into the analysis of this display. If we changed the design, we could put him more at ease when using this life-critical interface.

DoctorStrange-thisthingon.png

Initiating gesture

The initiating gesture doesn’t read as “set a savepoint.” This doesn’t show itself as a problem in this scene, but if the gesture did have some sort of semantic meaning, it would make it easier for Strange to recall and perform correctly. Maybe if his wrist twist transitioned from moving splayed fingers to his pointing with his index finger to his wrist…ok, that’s a little too on the nose, so maybe…toward the ground, it would help symbolize the here & now that is the savepoint. It would be easier for Strange to recall and feel assured that he’d done the right thing.

I have questions about the extents of the time loop effect. Is it the whole Dark Dimension? Is it also Earth? Is it the Universe? Is it just a sphere, like the other modes of the Eye? How does he set these? There’s not enough information in the movie to backworld this, but unless the answer is “it affects everything” there seems to be some variables missing in the initiating gesture.

Setpoint-active signal

But where the initiating gesture doesn’t appear to be a problem in the scene, the wrist-glance indicates that the display is. Note that, other than being on the left forearm instead of the right, the bands look identical to the ones in the Tibet and Hong Kong modes. (Compare the Tibet screenshot below.) If Strange is relying on the display to ensure that his savepoint was set, having it look identical is not as helpful as it would be if the visual was unique. “Wait,” he might think, “Am I in the right mode, here?

Eye-of-Agamoto10.png

In a redesign, I would select an animated display that was not a loop, but an indication that time was passing. It can’t be as literal as a clock of course. But something that used animation to suggest time was progressing linearly from a point. Maybe something like the binary clock from Mission to Mars (see below), rendered in the graphic language of the Eye. Maybe make it base-3 to seem not so technological.

binary_clock_10fps.gif

Seeing a display that is still, on invocation—that becomes animated upon initialization—would mean that all he has to do is glance to confirm the unique display is in motion. “Yes, it’s working. I’m in the Groundhog Day mode, and the savepoint is set.

The Museum of One Film

Years ago in grad school I heard a speaker tell of a possibly-imaginary museum called the Museum of One Painting. In that telling, the museum was a long hall. The current One Painting (they were occasionally switched out) was hung at the far end from the entrance. As you walked the length of it, to your left you would see paintings and exhibits of the things that had influenced the One Painting. Then at the end you would spend time with the One Painting. On your way out, to your left you could see paintings and other artworks that were influenced by the One Painting.

Fplan-matte.png

Ah yes, I believe this was made during Henry Hillinick’s Robbie the Robot phase.
Hat tip to the awesome Matte Shot.

I loved this museum concept. It was about depth of understanding. It provided context. It focused visits on building a shared understanding that you could discuss with other visitors, even if they’d gone a week before you. My kind of art museum. I fell in love with this concept pretty hard and began to believe in the intervening decades that it was just a fable, constructed by wishful museum theorists.

Nope. Today I searched for it, and found it. It’s real. It’s housed in a small building in Penza, Russia. The reality is a little different than how I had it described (or, rather, how I wrote it to memory), and I think it’s only in Russian (mine is a pittance), and given recent politics I’m not sure I’d be welcome there; but its beautiful core concept is intact. A deep dive into a single painting at a time.

Penza.png

Just a quick 9-hour drive from Moscow.

The whole reason I bring this up on the blog is because the awesome American cinema chain Alamo Drafthouse is doing something like this, but for film. And not just any film, but for the upcoming Blade Runner 2049. Their Road to Nowhere series examines dystopian films that influenced or were influenced by Blade Runner. Some you probably know and love. (Metropolis! Logan’s Run! The Fifth Element!) Some I’d never heard of but now want to. (1990: The Bronx Warriors! Hardware! Prayer of the Rollerboys!)

Road to Nowhere.png

I try not to be a gushing fanboy for anything on this blog, but I gotta hand it to the Drafthouse for this. This is my kind of film nerdery. If I was to run a film series, it would be just like this, only with some sci-fi interface analysis and redesign meetups thrown in for good measure. Just thrilled that it’s happening, and there’s a Drafthouse near me in San Francisco. (Sorry if there’s not one near you, but maybe there’s something similar?)

Anyway, I was not paid by anyone to write this. Just…just happy and nerding out. Hope to see you there.

The Hong Kong Mode (4 of 5)

In the prior three posts, I’ve discussed the goods-and-bads of the Eye of Agamotto in the Tibet mode. (I thought I could squeeze the Hong Kong and the Dark Dimension modes into one post, but turns out this one was just too long. keep reading. You’ll see.) In this post we examine a mode that looks like the Tibet mode, but is actually quite different.

Hong Kong mode

Near the film’s climax, Strange uses the Eye to reverse Kaecilius’ destruction of the Hong Kong Sanctum Sanctorum (and much of the surrounding cityscape). In this scene, Kaecilius leaps at Strange, and Strange “freezes” Kaecilius in midair with the saucer. It’s done more quickly, but similarly to how he “freezes” the apple into a controlled-time mode in Tibet.

HongKong-freeze-12fps.gif

But then we see something different, and it complicates everything. As Strange twists the saucer counterclockwise, the cityscape around him—not just Kaecilius—begins to reverse slowly. (And unlike in Tibet, the saucer keeps spinning clockwise underneath his hand.) Then the rate of reversal accelerates, and even continues in its reversal after Strange drops his gesture and engages in a fight with Kaecilius, who somehow escapes the reversing time stream to join Strange and Mordo in the “observer” time stream.

So in this mode, the saucer is working much more like a shuttle wheel with no snap-back feature.

A shuttle wheel, as you’ll recall from the first post, doesn’t specify an absolute value along a range like a jog dial does. A shuttle wheel indicates a direction and rate of change. A little to the left is slow reverse. Far to the left is fast reverse. Nearly all of the shuttle wheels we use in the real world have snap-back features, because if you were just going to leave it reversing and pay attention to something else, you might as well use another control to get to the absolute beginning, like a jog dial. But, since Strange is scrubbing an endless “video stream,” (that is, time), and he can pull people and things out of the manipulated-stream and into the observer-stream and do stuff, not having a snap-back makes sense.

For the Tibet mode I argued for a chapter ring to provide some context and information about the range of values he’s scrubbing. So for shuttling along the past in the Hong Kong mode, I don’t think a chapter ring or content overview makes sense, but it would help to know the following.

  • The rate of change
  • Direction of change
  • Shifted datetime
  • Timedate difference from when he started

In the scene that information is kind of obvious from the environment, so I can see the argument for not having it. But if he was in some largely-unchanging environment, like a panic room or an underground cave or a Sanctum Sanctorum, knowing that information would save him from letting the shuttle go too far and finding himself in the Ordovician. A “home” button might also help to quickly recover from mistakes. Adding these signals would also help distinguish the two modes. They work differently, so they should look different. As it stands, they look identical.

DoctorStrange-Tibet-v-HongKong.png

He still (probably) needs future branches

Can Strange scrub the future this way? We don’t see it in the movie. But if so, we have many of the same questions as the Tibet mode future scrubber: Which timeline are we viewing & how probable is it? What other probabilities exist and how does he compare them? This argues for the addition of the future branches from that design.

Selecting the mode

So how does Strange specify the jog dial or shuttle wheel mode?

One cop-out answer is a mental command from Strange. It’s a cop-out because if the Eye responds to mental commands, this whole design exercise is moot, and we’re here to critique, practice, and learn. Not only that, but physical interfaces are more cinemagenic, so better to make a concrete interaction for the film.

You might think we could modify the opening finger-tut (see the animated gif, below). But it turns out we need that for another reason: specifying the center and radius-of-effect.

DoctorStrange-tutting-comparison.gif

Center and radius-of-effect

In Tibet, the Eye appears to affect just an apple and a tome. But since we see it affecting a whole area in Hong Kong, let’s presume the Eye affects time in a sphere. For the apple and tome, it was affecting a small sphere that included the table, too, it’s just that table didn’t change in the spans of time we see. So if it works in spheres, how is the center and the radius of the sphere set?

Center

Let’s say the Eye does some simple gaze monitoring to find the salient object at his locus of attention. Then it can center the effect on the thing and automatically set the radius of effect to the thing’s size across likely-to-be scrubbed extents. In Tibet, it’s easy. Apple? Check. Tome? Check. In Hong Kong, he’s focusing on the Sanctum, and its image recognition is smart enough to understand the concept of “this building.”

Radius

But the Hong Kong radius stretches out beyond his line of sight, affecting something with a very vague visual and even conceptual definition, that is, “the wrecked neighborhood.” So auto-setting these variables wouldn’t work without reconceiving the Eye as a general artificial intelligence. That would have some massive repercussions throughout the diegesis, so let’s avoid that.

If it’s a manual control, how does he do it? Watch the animated gif above carefully and see he’s got two steps to the “turn Eye on” tut: opening the eye by making an eye shape, and after the aperture opens, spreading his hands apart, or kind of expanding the Eye. In Tibet that spreading motion is slow and close. In Hong it’s faster and farther. That’s enough evidence to say the spread*speed determines the radius. We run into the scales problem of apple-versus-neighborhood that we had in determining the time extents, but make it logarithmic and add some visual feedback and he should be able to pick arbitrary sizes with precision.

So…back to mode selection

So if we’re committing the “turn on” gesture to specifying the center-and-radius, the only other gesture left is the saucer creation. For a quick reminder, here’s how it works in Tibet.

Since the circle works pretty well for a jog dial, let’s leave this for Tibet mode. A contrasting but related gesture would be to have Strange hold his right hand flat, in a sagittal plane, with the palm facing to his left. (See an illustration, below.) Then he can tilt his hand inside the saucer to reverse or fast forward time, and withdraw his hand from the saucer graphic to leave time moving at the adjusted rate. Let the speed of the saucer indicate speed of change. To map to a clock, tilting to the left would reverse time, and tilting to the right would advance it.

How the datetime could be shown is an exercise for the reader.

The yank out

There’s one more function we see twice in the Hong Kong scene. Strange is able to pull Mordo and Wong from the reversing time stream by thrusting the saucer toward them. This is a goofy choice of a gesture that makes no semantic sense. It would make much more sense for Strange to keep his saucer hand extended, and use his left hand to pull them from the reversing stream.

DoctorStrange-yank-out.gif

Whew.

So one of the nice things about this movie interface, is that while it doesn’t hold up under the close scrutiny of this blog,  the interface to the Eye of Agamotto works while watching the film. Audience sees the apple happen, and gets that gestures + glowing green circle = adjusting time. For that, it works.

That said, we can see improvements that would not affect the script, would not require much more of the actors, and not add too much to post. It could be more consistent and believable.

But we’re not done yet. There’s one other function shown by the Eye of Agamotto when Strange takes it into the Dark Dimension, which is the final mode of the Eye, up next.

Tibet Mode Analysis: Representing the future (3 of 5)

A major problem with the use of the Eye is that it treats the past and the future similarly. But they’re not the same. The past is a long chain of arguably-knowable causes and effects. So, sure, we can imagine that as a movie to be scrubbed.

But the future? Not so much. Which brings us, briefly, to this dude.

pierre-simon-laplace.png

If we knew everything, Pierre-Simon Laplace argued in 1814, down to the state of every molecule, and we had a processor capable, we would be able to predict with perfect precision the events of the future. (You might think he’s talking about a computer or an AI, but in 1814 they used demons for their thought experiments.) In the two centuries since, there have been several major repudiations of Laplace’s demon. So let’s stick to the near-term, where there’s not one known future waiting to happen, but a set of probabilities. That means we have to rethink what the Eye shows when it lets Strange scrub the future.

Note that in the film, the “future” of the apple shown to Strange was just a likelihood, not a fact. The Eye shows it being eaten. In the actual events of the film, after the apple is set aside:

  • Strange repairs the tome
  • Mordo and Wong interrupt Strange
  • They take him into the next room for some exposition
  • The Hong Kong sanctum portal swings open
  • Kaecilius murders a redshirt
  • Kaecilius explodes Strange into the New York sanctum

Then for the next 50 minutes, The Masters of Mysticism are scrambling to save the world. I doubt any of them have time to while away in a library, there to discover an abandoned apple with a bite taken out of it, and decide—staphylococcus aureus be damned—a snack’s a snack. No, it’s safe to say the apple does not get eaten.

post-eye-no-apple.png

So the Eye gets the apple wrong, but it showed Strange that future as if it were a certainty. That’s a problem. Sure, when asked about the future, it ought to show something, but better would be to…

  • Indicate somewhere that what is being displayed is one of a set of possibilities
  • Provide options to understand the probability distribution among the set
  • Explore the alternates
  • Be notified when new data shifts the probability distribution or inserts new important possibilities

So how to display probabilities? There are lots of ways, but I am most fond of probability tree diagrams. In nerd parlance, this is a unidirectional graph where the nodes are states and the lines are labeled for probabilities. In regular language they look like sideways two-dimensional trees. See an example below from mathisfun.com. These diagrams seem to me a quick way to understand branching possibilities. (I couldn’t find any studies giving me more to work on than “seem to me”.)

probability-tree-coin2.png

In addition to being easy to understand, they afford visual manipulation. You can work branching lines around an existing design.

Now if we were actually working out a future-probabilities gestural scrubber attached to the Eye of Agamotto saucer, we’d have a whole host of things to get into next, like designing…

  1. A compact but informative display that signals the relative probabilities of each timeline
  2. The mechanism for opening that display so probabilities can be seen rather than read
  3. Labels so Strange wouldn’t have to hunt through all of them for the thing of interest (or some means of search)
  4. A selection process for picking the new timeline
  5. A comparison mode
  6. A means of collapsing the display to return to scrub mode
  7. A you-are-here signal in the display to indicate the current timeline

Which is a big set of design tasks for a hobbyist website. Fortunately for us, Strange only deals with a simple, probable (but wrong) scenario of the apple’s future as an illustration for the audience of what the Eye can do; and he only deals with the past of the tome. So while we could get into all of the above, it’s most expedient just to resolve the first one for the scene and tidy up the interface as it helps illustrate a well-thought-out and usable world.

Below I’ve drafted up an extension of my earlier conceptual diagram. I’ve added a tree to the future part of the chapter ring, using some dots to indicate the comparative likelihood of each branch. This could be made more compact, and might be good to put on a second z-axis layer to distinguish it from the saucer, but again: conceptual diagram.

Eye-of-Agamoto-tail.png

If this were implemented in the film, we would want to make sure that the probability tree begins to flicker right before Wong and Mordo shut him down, as a nod to the events happening off screen with Kaecilius that are changing those futures. This would give a clue that the Eye is smartly keeping track of real-world events and adjusting its predictions appropriately.

These changes would make the Eye more usable for Strange and smart as a model for us.

Eye-of-Agamoto-01_comp.png

Twist ending: This is a real problem we will have to solve

I skipped those design tasks for this comp, but we may not be able to avoid those problems forever. As it turns out, this is not (just) an idle, sci-fi problem. One of the promises of assistive AI is that it will be giving its humans advice, based on predictive algorithms, which will be a set of probabilistic scenarios. There may be an overwhelmingly likely next scenario, but there may also be several alternatives that users will need to explore and understand before deciding the best strategy. So, yeah, an exercise for the reader.

Wrapping up the Tibet Mode

So three posts is not the longest analysis I’ve done, bit it was a lot. In recap: Gestural time scrubbing seems like a natural interaction mapped well to analog clocks. The Eye’s saucer display is cool, but insufficient. We can help Strange much more by adding an events-based chapter ring detailing the facts of the past and the probabilities of the future.

Alas. We’re not done yet. As you’ll recall from the intro post, there are two other modes: The Hong Kong and Dark Dimension modes. Let’s next talk the Hong Kong mode, which is like the Tibet mode, but different.