Tag Archives: schematic

Browsing Files in 3D

by

0

Be forewarned—massive spoilers ahead. (The graphic shows the Millennium Falcon sporting a massive spoiler.)

What’s this all about?

The origin story here is that I wanted to review Hackers, a film I enjoy and Chris describes as “awesome/ly bad”. However, Hackers isn’t science fiction. Well, I could argue that it is set in an alternate reality where computer hackers are all physically attractive with fashionable tastes in music and clothing, but that isn’t enough. The film was set firmly in the present day (of 1995) and while the possibilities of computer hacking may be exaggerated for dramatic purposes, all the computers themselves are quite conventional for the time. (And therefore appear quaint and outdated today.)

With the glorious exception of the three dimensional file storage system on the “Gibson” mainframe. This fantastic combination of hardware and software was clearly science fiction then, and remains so today. While one futuristic element is not enough to justify a full review of Hackers, it did start us thinking. The film Jurassic Park also has a 3D file system navigator, which wasn’t covered in depth by either the book or the online review. And when Chris reached out to the website readers, they provided more examples of 3D file systems being added to otherwise mundane computer systems.

So what we have here is a review of a particular interface trope rather than an entire film or TV show: the three dimensional file browsing and navigation interface.

Scope

This review is specifically limited to interfaces that are recognisably file systems, where people search for and manipulate individual documents or programs. Cyberspace, a 3D representation of the Internet or World Wide Web, is too broad a topic, and better covered in individual reviews such as those for Ghost in the Shell and Johnny Mnemonic.

I also originally intended to only include non-science fiction films and shows but Jurassic Park is an exception. Jurassic Park has been reviewed, both in the book and on the website, but the 3D file system was a comparatively minor element. It is included here as a well known example for comparison.

The SciFiInterfaces readership also provided examples of research papers for 3D file system browsing and navigation—rather more numerous than actual production systems, even today. These will inform the reviews but not be discussed individually.

Because we are reviewing a topic, not a particular film or TV show, the usual plot summaries will be shortened to just those aspects that involve the 3D file system. As a bonus, we can also compare and contrast the different interfaces and how they are used. The worlds of Ghost in the Shell and Johnny Mnemonic are so different that it would be unfair to judge individual interfaces against each other, but for this review we are considering 3D file systems that have been grafted onto otherwise contemporary computer systems, and used by unaugmented human beings to perform tasks familiar to most computer users.

Sources

Having decided on our topic and scope, the properties for review are three films and one episode of a TV show.

Jurassic Park, 1993

“I know this!” and Jurassic Park is so well known that I assume that you do too. We will look specifically at the 3D file system that is used by Lex in the control room to reactivate the park systems.

Disclosure, 1994

This film about corporate infighting includes a virtual reality system, complete with headset, glove, laser trackers, and walking surface, which is used solely to look for particular files.

Hackers, 1995

As mentioned in the introduction this film revolves around the hacking of a Gibson mainframe, which has a file system that is both physically three dimensional and represented on computer screens as a 3D browser.

A bar chart. The x-axis is every year between and including 1902 to 2022. The y-axis, somewhat humorously, shows 2-place decimal values up to 1. Three bars at 1.00 appear at 1993, 1994, and 1995. One also appears in 2016, but has an arrow pointing back to the prior three with a label, “(But really, this one is referencing those.)”

All three of these films date from the 1990s, which seems to have been the high point for 3D file systems, both fictional and in real world research.

Community, season 6 episode 2, “Lawnmower Maintenance and Postnatal Care” (2016)

In this 21st century example, the Dean of a community college buys an elaborate virtual reality system. He spends some of his time within VR looking for, and then deleting, a particular file. 

Clockwise from top left: Jurassic Park (1993), Disclosure (1994), Hackers (1995), Community (2016)

And one that almost made it

File browsing in two dimensions is so well established in general-purpose computer interfaces that the metaphor can be used in other contexts. In the first Iron Man film, at around 52 minutes, Tony Stark is designing his new suit with an advanced 3D CAD system that uses volumetric projection and full body gesture recognition and tracking. When Tony wants to delete a part (the head) from the design, he picks it up and throws it into a trashcan.

Tony deletes a component from the design by dropping it into a trashcan. Iron Man (2008)

I’m familiar with a number of 2D and 3D design and drawing applications and in all of them a deleted part quietly vanishes. There’s no more need for visual effects than when we press the delete key while typing.

In the film, though, it is not Tony who needs to know that the part has been deleted, but the audience. We’ve already seen the design rendering moved from one computer to another with an arm swing, so if the head disappeared in response to a gesture, that could have been interpreted as it being emailed or otherwise sent somewhere else. The trashcan image and action makes it clear to the audience what is happening.

So, that’s the set of examples we’ll be using across this series of posts. But before we get into the fiction, in the next post we need to talk about how this same thing is handled in the real world.

Cyberspace: Beijing Hotel

by

1

After selecting its location from a map, Johnny is now in front of the virtual entrance to the hotel. The virtual Beijing has a new color scheme, mostly orange with some red.

jm-33-hotel-a

The “entrance” is another tetrahedral shape made from geometric blocks. It is actually another numeric keypad. Johnny taps the blocks to enter a sequence of numbers.

The tetrahedral keypad

jm-33-hotel-b

Note that there can be more than one digit within a block. I mentioned earlier that it can be difficult to “press” with precision in virtual reality due to the lack of tactile feedback. Looking closely, here the fingers of Johnny’s “hands” cast a shadow on the pyramid, making depth perception easier.

Something is wrong, and Johnny receives an electric shock.

jm-33-hotel-c

He reacts as if the shock is real, pulling his hands back and cursing.

In the 1980s and 1990s cyberpunk books such as Neuromancer and Hardwired and roleplaying games such as Cyberpunk and ShadowRun suggested that future virtual reality systems would be able to physically attack users, the dreaded “Black ICE”. While the more vigilant Internet copyright enforcers would probably be in favour, it seems unlikely that the liability lawyers at any computer manufacturer would allow a product that could electrocute users to be released, or that users would agree to put something like that on their hands. So this is most likely  just Johnny expressing the same frustration as a current day video gamer who loses a life in a first person shooter.

The last necessary step before being granted access is, for some reason, to reshape the pyramid.

jm-32-beijing-b-animated

Here the pyramid serves as a combination lock or puzzle as well as a keypad. It’s not obvious, but Johnny does make a small 3D rotating gesture on the entire pyramid before pulling and pushing blocks around. You can also see a second layer of structure underneath the moving shapes.

Is this an effective security system? Not really. Two-factor authentication systems rely both on knowingsomething, here a numeric code, and either havingsomething, such as a specific mobile phone or token generator, or beingsomeone, with a specific fingerprint. Reshaping the blocks is just a second thing the would-be user must know, and is just as vulnerable to being guessed as the numeric code. On the other hand, it might be enough to keep out simple-minded attacks that only try the first step.

The floorplan

The “interior” of the hotel site is first displayed as a flat plan view. This builds up incrementally, a transition known among VR developers since the film Tron came out as “rezzing up”. The completed plan then rotates into a 3D structure.

jm-34-inside-a-animated

We hear the voice feedback announce “General accounts selected” but don’t see how Johnny did this. A window expands out, and Johnny splits it in half to reveal some tabular data.

jm-34-inside-b-animated

The fax and phone records are displayed in a simple tabular view, which would not look out of place on any 1995 or indeed current day desktop computer spreadsheet. There’s no need to use 3D graphics for such this.

jm-34-inside-d

There are new interface elements here, overlaying the tabular data in pink. At the top we can read SEARCH > FAX CHARGES: FOUND. And on the right is a set of inscrutable numbers with headings GRID, LEVEL, MENU, and XYZ. This could be some orientation within the data, but it doesn’t make sense. In the lower-left we see a label for elevation, with data as “coordinates in sector 4.”

Below that a 9-key arrangement with arrow shapes. Perhaps this is a navigation aid for people using conventional 2D desktop interfaces rather than full virtual reality equipment, allowing them to move around by clicking the onscreen arrows or pressing the equivalent keys. If the keys are similar to those used in computer games, the up and down arrow keys move forward or backwards and the left and right keys rotate, assuming movement is predominantly in the horizontal plane. The other keys might be for banking or vertical movement.

Johnny searches for the outgoing fax. He does not use any graphical gestures for this, instead specifying the search date and time ranges by speaking. Words and operators are more precise than graphic symbols for this kind of database query, but typing on a virtual keyboard would be more awkward than speech.

When the particular table cell is found, he uses the fingertips of both hands to expand the contents, one of the standard gestures described in the Make It So book.

Not surprisingly for a Beijing hotel, the internal records are not in English. Johnny again uses a voice command to ask for translation.

jm-34-inside-e

The hotel record is just the metadata, not the actual images he’s looking for, suggesting that “fax” system is fully digital and the faxes themselves are treated like modern email messages and deleted once sent. The metadata does tell Johnny that the images were faxed to a online copyshop in Newark. Since it is network connected, Johnny can jump straight to it in cyberspace.

Brain Scanning

by

2

The second half of the film is all about retrieving the data from Johnny’s implant without the full set of access codes. Johnny needs to get the data downloaded soon or he will die from the “synaptic seepage” caused by squeezing 320G of data into a system with 160G capacity. The bad guys would prefer to remove his head and cryogenically freeze it, allowing them to take their time over retrieval.

1 of 3: Spider’s Scanners

The implant cable interface won’t allow access to the data without the codes. To bypass this protection requires three increasingly complicated brain scanners, two of them medical systems and the final a LoTek hacking device. Although the implant stores data, not human memories, all of these brain scanners work in the same way as the Non-invasive, “Reading from the brain” interfaces described in Chapter 7 of Make It So.

The first system is owned by Spider, a Newark body modification
specialist. Johnny sits in a chair, with an open metal framework
surrounding his head. There’s a bright strobing light, switching on
and off several times a second.

jm-20-spider-scan-a

Nearby a monitor shows a large rotating image of his head and skull, and three smaller images on the left labelled as Scans 1 to 3.

jm-20-spider-scan-b

The largest image resembles a current-day MRI or CT display. It is being drawn on a regular flat 2D display rather than as a 3D holographic type projection, so does not qualify as a volumetric projection even though a current day computer graphics programmer might call it such. The topmost Scan 1 is the head viewed from above in the same rendering style. Scan 2 in the middle shows a bright spot around the implant, and Scan 3 shows a circuit board, presumably the implant itself. The background is is blue, which so far has been common but not as predominant as it is in other science fiction interfaces. Chris suggests  this is because blue LEDs were not common in 1995, so the physical lights we see are red and green and likewise the onscreen graphics use many bright colors.

jm-20-spider-scan-c

Occasionally a purple bar slides across the main image. It perhaps represents some kind of processing update, but since the image is already rotating, that seems superfluous. At one point the color of the main image changes to red, with a matching red sliding bar, but we don’t know why. All the smaller images rotate or flash regularly, with faint ticking sounds as they do.

From this system, Spider is able to tell Johnny that there is a problem with his implant and it must be painful. (Understandably, Johnny is not impressed with this less than helpful diagnosis.) Unlike either the scanner at Newark Airport or the LoTek binoculars, there are no obvious messages or indicators providing this information. But this is a specialised piece of medical technology rather than a public access system, so presumably Spider has sufficient expertise to interpret the displays without needing large popup text.

2 of 3: Hospital Scanner

Spider takes Johnny to a hospital for a more thorough scan. Here the first step is attaching a black flexible strip with various cables around his head. His implant cable is also connected.

jm-21-hospital-scan-b

There isn’t a clear shot of the entire system, but behind Johnny is a CRT monitor and to his left, our right, is a bank of displays that look like electronic oscilloscopes. Since embedded body electronics are common in the world of Johnny Mnemonic, that is probably exactly what they are intended to be. Spider adjusts some controls on these.

jm-21-hospital-scan-c

The oscilloscopes show no text, just green lines and shapes. The CRT behind Johnny is now showing the same head image that we saw at the end of the previous scan.

jm-21-hospital-scan-d

In front of the oscilloscopes is a PC keyboard from the 1990s. In 2021 this will look even older, but this entire hospital is portrayed as a shoestring operation relying on donations and salvage. Spider types on the keyboard, and the CRT changes to show a lot of scrolling text.

jm-21-hospital-scan-e

This is enough for Spider to announce that the “data” is the cure for NAS, the world wide epidemic disease that Jane is showing symptoms of. Again it’s not clear how he can determine this, as the data is still protected by the access codes. Perhaps the scrolling text is unencrypted metadata in the implant that is more easily retrieved. Given the apparent hazardous life of a mnemonic courier, it would make sense to attach the equivalent of a sticky label to the implant, briefly describing the contents and who they should be delivered to.

(This is also the point where one has to ask why this valuable data is encrypted and protected to begin with. Using a mnemonic courier for distribution makes sense, to avoid content filters on the Internet. But now the data is here in Newark, with the intended recipients, so why is it so hard to get at? The best answer I can think of is that the scientists wanted to ensure that the mnemonic courier couldn’t keep the data for themselves and sell it to the highest bidder.)

The third of the three brain interfaces warrants its own post, coming up next. 

Iron Man HUD: Just the functions

by

1

In the last post we went over the Iron HUD components. There is a great deal to say about the interactions and interface, but let’s just take a moment to recount everything that the HUD does over the Iron Man movies and The Avengers. Keep in mind that just as there are many iterations of the suit, there can be many iterations of the HUD, but since it’s largely display software controlled by JARVIS, the functions can very easily move between exosuits.

Gauges

Along the bottom of the HUD are some small gauges, which, though they change iconography across the properties, are consistently present.

IronMan1_HUD07

For the most part they persist as tiny icons and thereby hard to read, but when the suit reboots in a high-altitude freefall, we get to see giant versions of them, and can read that they are:

A digital hologram of a humanoid figure surrounded by a circular interface, set against a dark background with city lights visible below.
Overhead view of a city at night, featuring a digital targeting crosshair overlay against a backdrop of illuminated buildings and streets.
A digital circular interface displaying various data points and graphical elements against a dark urban background at night.
A futuristic HUD display showcasing data and metrics over a cityscape at night.
A futuristic holographic display showing a digital globe with concentric circles, set against a dark background suggesting movement through space.
IronMan1_HUD13
Tony can, at a glance or request, summon more detail for any of the gauges.
IronMan1_HUD12
Even different visualizations of similar information.

Object Recognition

In the 1st-person view we see that the HUD has a separate map in the lower-left, and object recognition/awareness,

IronMan1_HUD10
IronMan1_HUD11
In the 2nd-person view, we see even more layers of information about the identified objects, floating closer to tony’s point of view.

Situational

Most of the HUD functions we see, though, are situational, brought up for Tony’s attention when JARVIS believes they are needed, or when Tony requests them. Following are screenshots that illustrate a moment when the situational function appeared. 

Iron Man

Close-up of a man's face with a focused expression, surrounded by futuristic digital interfaces and data displays.
Close-up of a man's face with intense expression, surrounded by holographic interface displaying status information.
Cockpit view displaying a digital interface with readings and data, featuring the moon in the background and a depiction of the SR-71 Blackbird aircraft.
Close-up of a man's face reflecting digital interfaces and data, with intense focus and a slight expression of concern.
Close-up of a man's face looking intently at a futuristic, holographic interface with various digital elements.
Close-up of a face with a focused expression, surrounded by digital interfaces and holographic displays.
Close-up of a man's face with a focused expression, interacting with holographic digital interfaces and data displays.
Close-up view of a spacecraft dashboard displaying navigation instruments alongside a visible moon in a starry background.

Iron Man 2

A futuristic interior setting with a humanoid figure in metallic armor, high-tech holographic displays on the walls, and modern furniture.
Close-up of a man's face reflecting digital interfaces and screens in a space helmet, showcasing high-tech graphics and imagery.
Close-up of a man’s face, focused on a high-tech interface with map graphics and data displays in the background.
Futuristic urban scene featuring holographic displays, including a globe and various digital metrics, amid a bustling crowd at night.
A man in a futuristic suit appears in a dark underwater environment, surrounded by digital interface elements and targeting graphics.

Iron Man 3

A robotic figure with a helmet is seen near a screen displaying a red flashlight labeled 'OFFLINE' and a message indicating 'SYSTEM FAILURE'.
Close-up of a man's face with a bloodied injury on his nose, appearing distressed, set against a futuristic display interface with holographic elements.
Close-up of a man's face with a serious expression, featuring a blood streak on his nose and digital interface elements in the background indicating system failure.
A futuristic robotic suit, resembling Iron Patriot, is depicted flying. A holographic display of the suit's specifications and details is visible in the background.
A futuristic digital interface displaying a target acquisition screen with various metrics and figures, surrounded by figures falling through clouds.
A high-tech interface overlay displays target indicators on ocean water, with a small boat in the distance and a city skyline in the background.
Close-up of a man's face with intense blue eyes, showing signs of injury and a digital interface in the background, displaying data and a news image.

The Avengers

Close-up of a male character intently looking at a holographic interface, showcasing various digital readouts and graphics.
Close-up of a man's face with digital interface elements displayed around him, showing critical velocity and damage information.
A futuristic digital interface displaying a stylized 3D model composed of sharp, fragmented shapes, with red highlights and technical readouts in a dark background.
Close-up of a man's face with holographic displays and high-tech interfaces surrounding him.
Close-up of a man with a serious expression, gazing at a holographic display showing a woman's image and audio waveform, with high-tech interfaces visible in the background.
A futuristic heads-up display showing a view of a bridge over water with overlays of mission data, including a rocket and explosive indicators.
Close-up of a man's face with intense expression, analyzing a futuristic holographic display with glowing red graphics.

Some of these illustrate why I argue that JARVIS is the superhero, and Tony just the onboard manager, but rather than reverse engineering any particular function, for this post it is enough to document them and note that only the optical zoom seems to be an interactive function. This raises questions of how he initiated the mode and how he escapes the mode, but since we don’t see the mechanisms of control, it’s entirely arguable that JARVIS is just  being his usual helpful self again.

Next up in the Iron HUD series: Let’s dive deeper into the first-person view.