Tag Archives: hologram

Dr. Strange’s augmented reality surgical assistant

by

0

We’re actually done with all of the artifacts from Doctor Strange. But there’s one last kind-of interface that’s worth talking about, and that’s when Strange assists with surgery on his own body.

After being shot with a soul-arrow by the zealot, Strange is in bad shape. He needs medical attention. He recovers his sling ring and creates a portal to the emergency room where he once worked. Stumbling with the pain, he manages to find Dr. Palmer and tell her he has a cardiac tamponade. They head to the operating theater and get Strange on the table.

DoctorStrange_AR_ER_assistant-02.png

When Strange passes out, his “spirit” is ejected from his body as an astral projection. Once he realizes what’s happened, he gathers his wits and turns to observe the procedure.

DoctorStrange_AR_ER_assistant-05.png

When Dr. Palmer approaches his body with a pericardiocentesis needle, Strange manifests so she can sense him and recommends that she aim “just a little higher.” At first she is understandably scared, but once he explains what’s happening, she gets back to business, and he acts as a virtual coach.

DoctorStrange_AR_ER_assistant-08.png

In this role he points at the place she should insert the needle, and illuminates the chest cavity from within so she can kind of see the organ she’s targeting and the surrounding tissue. She asks him, “What were you stabbed with?” and he must confess, “I don’t know.”

DoctorStrange_AR_ER_assistant-11.png

Things go off the rails when the zealot who stabbed him shows up also as an astral projection and begins to fight Strange, but that’s where we can leave off the narrative and focus on everything up to this point as an interface.

Imagine with me, if you will, that this is not magic, but a kind of augmented reality available to the doctor. Strange is an unusual character in that he is both one of the world’s great surgeons and the patient in the scene, so let’s tease apart each.

An augmented reality coach

Realize that Dr. Palmer is getting assistance from one of the world’s greatest surgeons, rendered as a volumetric projection (“hologram” in vernacular). She can talk to him as if he was there to get his advice, and, I presume, even dismiss him if she believes he was wrong. Wouldn’t doctors working in new domains relish the opportunity to get advice from experts until they they have built their own mastery?

Two notes to extend this idea.

In the spirit of evidence-based medicine and big idea, we must admit that it would be better to have diagnoses and advice based on the entirety of the medical record and current, ethical best practices, not just one individual expert. But if an individual doctor prefers to have that information delivered through an avatar of a favored mentor, why not?

The second note to anyone thinking of this as a real world model for an AR assistant: I would expect a fully realized solution to include augmentations other than just a human, of course, such as ideal angles for incisions, depth meters, and life signs.

A (crude) body visualization

One of the challenges surgeons have when working with internal damage is that the body is largely opaque. They have to use visualization tools like radiographs and (very) educated guesses to diagnose and treat what’s going on inside these fleshy boxes of ours. How awesome that the AR coach can help illuminate (in both senses) the body to help Dr. Palmer perform the procedure correctly?

DoctorStrange_AR_ER_assistant-13.png

Admittedly, what we see in Doctor Strange is a crude version. This same x-ray vision appeared with more clarity and higher resolution in two other films, as cited in the Medical chapter of Make It So. In Lost in Space, the medical table projects a real-time volumetric scan of the organs inside Judy’s body into the eyes of the observers.

7809592870_cf8f3fdccf_o.jpg

In Chrysalis, Dr. Bruger sees a volumetric display of the patient on which she is teleoperating.

7809609070_b65b6fb936_o.jpg

But despite its low-resolution, I wanted to draw it out as another awesome and somewhat subtle part of the way this AR assistant helps the doctor.

A queryable patient avatar

Lastly, consider that Dr. Palmer is able to ask her patient what happened to him. Of course in the real world passed out patients aren’t able to answer questions, but of course understanding the events that led to a crisis are important. I can imagine several sci-fi ways that this information might be retrievable from the world.

  • Trace evidence on the patient’s body: High-resolution sensors throughout the operating theater could have automatically run forensic analysis on the patient the moment they entered the room to determine type of wound and likely causes, such as  microscopic detection of soot in entrance wounds.
  • Environmental sensors: If the accident happened in a place with sensors that are queryable, then the assistant could look at video footage, or listen in to microphones in the environment to help piece together what happened. Of course the notion of a queryable technological panoptican has massive privacy issues which cannot be overlooked, but if the information is available to medical professionals, it would be tragic to ignore it in genuine crises.
  • Human witnesses can provide informative narratives. Witness and first responders may be on record already. But in looking at the environmental sensors, the assistant might be able to instantly reach out to those who have not. Imagine one of these witness, shaken by the event he saw, on a commute home. His phone buzzes and it is the assistant saying, “Hello, Mr. Mackinnon. Records indicate that you were witness to a violent crime today, and your account of the event is needed for the victim, who is currently in surgery. Can you take a moment to answer some questions?”
  • Patient preferences should be automatically exposed and incorporated via the assistant as well. If the patient was a Jehovah’s Witnesses, for instance, then their desire not to have a blood transfusion should be raised in whatever form the assistant takes.

An surgical assistant could automatically query all of these sources to make a hypothesis of what happened and advise the procedure. This could be available doctor for the asking, volunteered by the assistant at a lull in more critical action, or offered by the assistant as a preventative. I suspect it’s more likely the doctor would ask the assistant than the patient, e.g. “OK, ERbot, what happened to this guy?” but if the doctor prefers, she should be able ask in the second person, as Dr. Palmer does in the scene, and the system should reply appropriately.

Sure, in this context, it’s magic, but since we can imagine how it could be done with technology, this scene gives us a very dense set of inspirational ideas for the future of surgical assistants.

The answer does not program

by

9

LogansRun224

Logan’s life is changed when he surrenders an ankh found on a particular runner. Instead being asked to identify, the central computer merely stays quiet a long while as it scans the objects. Then its lights shut off, and Logan has a discussion with the computer he has never had before.

The computer asks him to “approach and identify.” The computer gives him, by name, explicit instructions to sit facing the screen. Lights below the seat illuminate. He identifies in this chair by positioning his lifeclock in a recess in the chair’s arm, and a light above him illuminates. Then a conversation ensues between Logan and the computer.

LogansRun113

The computer communicates through a combination of voice and screen, on which it shows blue text and occasional illustrative shapes. The computer’s voice is emotionless and soothing. For the most part it speaks in complete sentences. In contrast, Logan’s responses are stilted and constrained, saying “negative” instead of “no,” and prefacing all questions with the word, “Question,” as in, “Question: What is it?”

On the one hand it’s linguistically sophisticated

Speech recognition and generation would not have a commercially released product for four years after the release of Logan’s Run, but there is an odd inconsistency here even for those unfamiliar with the actual constraints of the technology. The computer is sophisticated enough to generate speech with demonstrative pronouns, referring to the picture of the ankh as “this object” and the label as “that is the name of the object.” It can even communicate with pragmatic meaning. When Logan says,

“Question: Nobody reached renewal,”

…and receives nothing but silence, the computer doesn’t object to the fact that his question is not a question. It infers the most reasonable interpretation, as we see when Logan is cut off during his following objection by the computer’s saying,…

“The question has been answered.”

Despite these linguistic sophistications, it cannot parse anything but the most awkwardly structured inputs? Sadly, this is just an introduction to the silliness that is this interface.

Logan undergoes procedure “033-03,” in which his lifeclock is artificially set to blinking. He is then instructed to become a runner himself and discover where “sanctuary” is. After his adventure in the outside performing the assignment he was forced to accept, he is brought in as a prisoner. The computer traps him in a ring of bars demanding to know the location of sanctuary. Logan reports (correctly) that Santuary doesn’t exist.

LogansRun206
LogansRun205
hologram

On the other hand, it explodes

This freaks the computer out. Seriously. Now, the crazy thing is that the computer actually understands Logan’s answer, because it comments on it. It says, “Unacceptable. The answer does not program [sic].” That means that it’s not a data-type error, as if it got the wrong kind of input. No, the thing heard what Logan was saying. It’s just unsatisfied, and the programmer decided that the best response to dissatisfaction was to engage the heretofore unused red and green pixels in the display, randomly delete letters from the text—and explode. That’s right. He decided that in addition to the Dissatisfaction() subroutine calling the FreakOut(Seriously) subroutine, the FreakOut(Seriously) subroutine in its turn calls Explode(Yourself), Release(The Prisoner), and the WhileYoureAtItRuinAllStructuralIntegrityoftheSurroundingArcitecture() subroutines.

LogansRun221

Frankly, if this is the kind of coding that this entire society was built upon, this whole social collapse thing was less deep social commentary and really just a matter of technical debt.

LogansRun223
LogansRun225
LogansRun226
LogansRun227

Technical. Debt.

The plastic educator

by

0

Dr. Morbius introduces the Krell “plastic educator,” saying, ““As far as I can make out, they used it to condition and test their young, in much the same way as we once employed finger painting among our kindergarten children.””

Morbius grasps the educator’s head mount.

The device is a station at which the learner sits. There is a large dashboard before him, in turn before a space enclosed in a tetrahedral encasement of plastic. To his right is a large column made of plastic with red and yellow graduations running up the side. Inside the column is a strange shape like a lathed accordion, terminating in a pulsing ring that indicates a level against the graduations. An arced panel hangs from the ceiling with other printed graduations with lines of light above and below. Blue neon squiggles blink randomly along the walls.

Morbius demonstrates proper placement of the educator interface.

To activate the station, the learner grasps a pair of curved metal arms, which are connected at a hinged base and tipped with crystal orbs. He leans forward, rests his forehead on a third arm, and pulls the pair of arms to rest on his temples. He turns a pair of dials on the dashboard before him, and the crystal orbs on all three arms glow, indicating that the headset is operational.

Morbius points to the intelligence indicator.

Adams and Doc try to guage their own IQs.

The device’’s immediate result is that the accordion shape inside the column rises such that the lit ring indicates the intelligence of the user. (To Adams’ and Doc’’s dismay, their readings are much lower than Morbius’.)

With the press of a lever Morbius manifests a thought visually.

The primary function of the device is for the user to make a thought of theirs manifest in the tetrahedral space. The user concentrates on the thing, and then pulls a lever at the base of the headset. A red ring at the base of the headset illuminates, and a material appears above a pedestal at the base of the tetrahedron. By concentrating, the user shapes this material into the desired thing. Morbius shapes it into an image of Alta. The image is a scaled, translucent, volumetric display of Alta, which moves and smiles just as she would.

The projection ceases immediately when the mechanism is removed.

To stop using the device and shut down the projection, the learner simply lifts the lever and removes the headset from contact, and the orbs, the red ring, and the volumetric projection all fade within moments.

Finished with his demonstration, Morbius turns the educator off.

Turning the dashboard off requires a user to turn two free-spinning dials that sit to each side of the headset inwards. The lights of the dashboard fade.