How can direct manipulation work on objects that are too large to be directly manipulated?
Sci-fi University critically examines interfaces in sci-fi that illustrate core design concepts. In this 3:30 minute episode, Christopher discusses how the interfaces of Ghost in the Shell introduces synecdoche to our gestural language.
If you know someone who likes anime, and is interested in natural user interfaces—especially gesture—please share this video with them.
With the reviews of Oblivion behind us, and The Avengers: Age of Ultron upon us in a matter of days, I thought it would be good to review the movie that canonized Joss Whedon into Hollywood sainthood so hard they had to retcon Catholicism into the timeline so this joke could happen.
Here it is, a supercut of every user interface in The Avengers (2013).
Boy there are some amazing interfaces in there. Got any favorites?
According to the director, Oblivion is “a daylight science fiction film with a kind of Twilight Zone story,” a callback to pre-Star Wars, 1970’s lonely man sci-fi set against a huge backdrop. (Read the full interview by Germain Lussier on /Film for more.) Certainly, it’s more visually-satisfying thing than intellectually-satisfying thing, but fortunately that same thing does not play out in the interfaces.
Sci: B (3 of 4) How believable are the interfaces?
One of the great strengths of the interfaces are their deep ties to the diegesis. There’s little fuidgetry, little that could be generically lifted and placed in another film. It’s what we used to call site-specific in design school and that’s a good thing for believability.
See how in Vika’s desktop the sections of interface contain things she has to monitor: Land, hydrorigs, drones, the Tet’s orbital position. Most of the interfaces in the film are this considered.
And then…then…then there’s the narrative-shortcut trope of the oh-by-the-way faster-than-light communication system that would have meant a much more advanced (and more defended?) world for the Tet to encounter in the first place.
So, some dings.
Fi: A (4 of 4) How well do the interfaces inform the narrative of the story?
This is where Oblivion’s interfaces really shine. They’re gorgeously realized with a rich stylistic and motion language. But moreso IMHO some of the apparent “problems” with the interfaces actually tell of the deep deception by the Tet. It’s core to telling that central story, and partly told through the interfaces.
Jack’s bike doesn’t make a lot of sense in the diegesis except that it is a perfect outlet for his sense of “freedom.” Tet’s thinking: Whew. Glad he has that outlet.
Other narrative aspects of the interfaces like the drone programmer help underscore the drones as aggressive, suspect, and alien, rather than defensive human measures.
I’d add a + to that A if the drones hadn’t been designed to look evil and menacing. Had they been more Hello Kitty and less Galactic Empire, Jack might have been less suspicious.
If it needs to be said: Not actually from Oblivion. Maybe the reboot.
Interfaces: B (3 of 4) How well do the interfaces equip the characters to achieve their goals?
The centerpiece of the film is Vika’s desktop. It’s her command and control center workstation that enables her to manage the strategy to Jack’s tactics, and even rest her teacup as she works. The most commonly accessed bits are in easy reach, and the display-only information is turned vertically for her like a clock on the wall.
It has a few ergonomic problems, like angling its displays away from her observational sphere (just for a teacup?) It doesn’t equip her for crisis conversations like it should. Some of its interactions are inconsistent. It sometimes makes her hunt for information rather than leading her there. But, all in all, a nice dashboard for her task.
There are other interesting bits, like the situationally-shaped reticle, the breakfast table that allows for sitrep breakfasts, and well-mapped Odysseycontrols that imply a bit of agentive support.
There are some usability problems throughout, or it would have fared better, but overall a good show.
Final Grade B+ (10 of 12), MUST-SEE
All told, these interfaces are rich and powerful and embody solid modern thinking about visual styling, motion design, gestural interaction, and heads-up-displays. Big props to that pro gmunk for his work (and keep an eye out for an interview with him about his work on the film soon.)
And may I send out a special shout-out to the guest bloggers for their excellent insights and write-ups: Clayton, Aleatha, Heath, and Maximilion. They did great and I’m very glad that at least four other people in the world know how much effort goes into providing these in-depth interface analyses. Let’s hope we hear from more about them on this blog in the future.
Pitch time: Learn more lessons about gestural interfaces, heads-up-displays, and other interface concepts from a vast survey of science fiction movies and television programs in the book I co-authored with Nathan Shedroff, Make It So: Interaction Design Lessons from Science Fiction.
Today, a post touching on some of the ethical issues at hand with drone technology.
Much of the debate today and in science fiction about drone technologies rightly focuses on ethics. To start, it is valuable to remember that drones are merely a technology like any other. While the technology’s roots have been driven by military research and military applications, like, say, the internet, the examples in the prior posts demonstrate that the technology can be so much more. But of course it’s not that simple.
Hang on, it’s going to get nerdy. But that’s why you come to this blog, innit?
Where drones become particularly challenging to assess in an ethical context is in their blurring of the lines of agency and accountability. As such, we must consider the ethics of the relationship between user/creator and the technology/device itself. A gun owner doesn’t worry about the gun…itself…turning on him or her. However, in Oblivion, for instance, Tom Cruise’s character flies and repairs the Predator-esque drones but then has them turn on him when their sensors see him as a threat.
While not obviously not an autonomous strong artificial intelligence, a real-world drone alternates between being operated manually and autonomously. Even a hobbyist with a commercial quadcopter can feel this awkward transition when they switch from flying their drone like an RC plane to preprogramming it to fly a pattern or take photos. This transition in agency has serious implications.
Anonymity
When you see someone standing in park, looking at a handheld control or staring up into the sky at their quadcopter buzzing around, the actions of that device are attributed to them. But seeing a drone flying without a clear operator in sight gives any observer a bit of the creeps. Science fiction’s focus on military drones has meant that the depictions can bypass questions about who owns and operates them—it is always the state or the military. But as consumer drones become increasingly available it will become unclear to whom or what we can attribute the actions of a drone. Just this year there was serious public concern when drones were spotted flying around historical landmarks in Paris because their control was entirely anonymous. Should drones have physical or digital identification to service a “license plate” of sorts that could link any actions of the drone to its owner? Most probably. But the bad guys will just yank them off.
The author (in blue) and Chris Noessel (in black), at InfoCamp.
Gap between agent and outcome
Many researchers have explored the difference between online behaviour and in-person behavior. (There’s a huge body of research here. Contact me if you want more information.) People have a lot less problem typing a vitriolic tweet when they don’t have to face the actual impact of that tweet on the receiver. Will drones have a similar effect? Unlike a car, where the driver is physically within the device and operating it by hand and foot, a drone might be operated a distance of hundreds of feet (or even thousands of miles, for military drones). Will this physical distance, mediated by a handheld controller, a computer interface, or a smartphone application change how the operator behaves? For instance, research has found that these operators, thousands of miles away from combat and working with a semi autonomous technology, are in fact at risk for PTSD. They still feel connected and in enough control of the drone that its effects have a significant impact on their mental health.
However, as drones become more ubiquitous, their applications become more diverse (and mundane), and their agency increases, will this connection remain as strong? If Amazon has tens of thousands of drones flying from house to house, do the technicians managing their flight paths feel the ethical implications of one crashing through a window the same as if they had accidentally knocked a baseball through? If I own a drone and program it to pick up milk from the store, do I feel fully responsible for its behaviour as part of that flight pattern Or does the physical distance, the intangible interface and the mediating technology (the software and hardware purchased from a drone company) disassociate the agent from the effect of the technology?Just imagine the court cases.
From Breaking Defense: As drone operations increase, the military is researching effective interfaces to support human operators
What are the ethical consequences of the different levels of agency an operator can provide to the drone? What are the moral consequences of increased anonymity in the use of these drone technologies? The U.S. military designs computer interfaces to help its drone pilots make effective decisions to achieve mission targets. Can science fiction propose designs, interfaces and experiences that help users of drone technologies achieve ethical missions?
Come on, sci-fi. Show the way.
People don’t know what to make of this technology. It’s currently the domain of the military, big technology companies, a few startups, and a hobbyist community generally ignored outside of beautiful, drone-filmed YouTube videos. Science fiction is a valuable (and enjoyable) tool for understanding technology and envisioning implications of new technologies. Science fiction should be pushing the limits of how drone technologies will change our world, not just exaggerating today’s worst applications.
As journalist and robotics researcher P.W. Singer puts it in his TED talk, “As exciting as a drone for search-and-rescue is, or environmental monitoring or filming your kids playing soccer, that’s still doing surveillance. It’s what comes next, using it in ways that no one’s yet imagined, that’s where the boom will be.”
In chats with friends and followers of the blog about sci-fi Drone Week, folks seem interested in coming up with a definitive list of sci-fi drones in movies and TV shows. While we might get there eventually by reviewing the movies and TV shows in which they appear, let’s beat that to the punch by creating the list FROM OUR MINDS.
Criteria:
In a sci-fi movie or television show
Is not simply a representation of a real-world drone
Is mobile (in the air, on the ground, or in space)
Appears (or defined as) technolgoical, not biological
Does not have a sentient controller/pilot aboard
Does not possess strong/general artificial intelligence
Either automous or remotely-controlled
Here’s what I’ve collected so far. Add more in the comments if you think of them.
Captain America: The Winter Soldier’s city-sized drones (2014)
Chappie (2015) (aerial-capable, but mostly ground)
Ground
The floor-sweeping robots from The Fifth Element (1995)
The Robot from the Lost in Space movie (1998), which Will could remotely control
The Spyders of Minority Report (2002)
The remotely-controlled robots in Surrogates (2009)
Microbots from Big Hero 6 (2014) (a swarm of drones)
Thanks to the following suggestors for the initial list: Kelley Strang, John Danuil, Devin Hartnett, Derek Eclavea, Lane Bourn, Wally Pfingsten & kedamono.
Almost but not quite
Some suggestions seem like they would be perfect candidates, but for some reason skim the definition.
Bit from Tron (1982) may have been limited to yes or no answers, but was an AI
Machine gun drones from the deleted scenes of Aliens (1986) can only swivel, not move
Dreadnought from Star Trek: Voyager (1995)(VOY, Dreadnought) it’s AI
Jarvis from the cinematic Iron Man/Avengers diegesis, also strong AI
Keep them coming in the comments. What did we forget?
Chris: Oh my drone it’s DRONE WEEK! Wait…what’s drone week?
Recently I was invited to the InfoCamp unConference at Berkeley where among the awesome and inspiring presentations, I sat in on Peter Swigert’s workshop on drones. Since the blog was deep in Oblivion, Pete and I agreed to coauthor a series of posts on this phenomenon, and also to set the record a little more straight for sci-fi fans and authors on the real-world state of drones.
Today, a post on some cool and totally not evil speculative uses of drones.
While drones are being used for positive purposes already, there will undoubtedly be myriad new applications as the technology develops and more people engage with its possibilities and implications. A few options include:
Voting drones
While purely online voting seems a long way off in the United States, drones could provide a physical link to voters but remove the logistical challenges of getting time off to travel to a polling station (a tactic that suppresses voting turnout and wastes time and resources.) Drones could go from house to house, authenticate by taking a photo of an individual’s face, their ID, and even their thumbprint, and a citizen could place their vote directly into the drone.
…much to the suppressors’ night terrors.
Weather management drones
With climate change likely to cause increasing challenges in weather, drones could be used as safe, effective tools for weather management. For instance, drones could seed clouds to promote rain. Drones equipped with weather sensors searching for the exact right place to release their payload could be more accurate and effective than current rocket based solutions. A cloud of drones with small parabolic lenses could cool an area by reflecting light away or warm one by concentrating it.
Biological replacement drones
Could drones replace certain species in the ecosystem? The collapse of bee communities in many parts of the world has been a major threat for agriculture and, if it needs to be said, most of human life on the planet. Should the worst happen to our bee friends in the future, could micro-drones serve the some pollination function as bees? In places where keystone species have gone extinct or can’t be maintained, could a drone be developed to automatically serve the same function? For instance, elephants knock down trees and create clearings in certain patterns, facilitating the transition from jungle to grassland. A drone could fly continuously, looking for patterns in the landscape or specific trees that an elephant would normally knock down, and either mark the trees for human removal or be constructed to damage the tree itself. Could these patterns and behaviors be used in terraforming new planets as well?
Little Johnny, it’s time you knew about the birds and the drones.
Avalanche prevention drones
Drones could scout avalanche prone areas and use computer vision and snow sampling to identify possible avalanches, and bring their own explosive payload to detonate preventative avalanches. (A practice done—dangerously—by humans today.) Backcountry skiers could rent time on a resort or park management’s drone to get a first person view of the terrain in real time before hitting the slopes.
Amber alert drones
Drones could be trained in facial recognition or even smell tracking (wouldn’t a bloodhound be more effective if it could fly and smell from the air?) to search for missing children. They could also serve as a notification system as they search for the child, broadcasting real time information to both investigators and citizens in the area.
Musical performance drones
Drones could provide on demand musical performances. No bandshell in a park? Just fly in some drones with speakers; the drones could align themselves appropriately to get the best acoustics for the setting.
“Well, mayhaps something a little less…dubsteppy?”
While some ideas may seem absurd (and Chris’ comps up there turn that up to 11), science fiction can provide an interesting testing ground for what these systems might look like if implemented. What are the implications of these ideas? What impact would they have on society? On our last Drone Week post tomorrow we’ll discuss some of the ethical considerations underneath all of this.
Chris: Day 2 of mighty mighty DRONE WEEK! Wait…what’s drone week?
Recently I was invited to the InfoCamp unConference at Berkeley where among the awesome and inspiring presentations, I sat in on Peter Swigert’s workshop on drones. Since the blog was deep in Oblivion, Pete and I agreed to coauthor a series of posts on this phenomenon, and also to set the record a little more straight for sci-fi fans and authors on the real-world state of drones.
Today Drone Week continues with the not-so-scary world of actual drones.
Delivery drones
While capitalism is a neutral force at best, the speculative drones that Amazon and Google are working on stand to make delivery faster and more direct. While these projects are undoubtedly driven by possible profits, Google suggests that rapid delivery by drone will also have significant social benefits. Astro Teller, director at Google X, suggests that on-demand drop off and pick up we of goods will let us need to own less. “It would help move us from an ownership society to an access society. We would have more of a community feel to the things in our lives. And what if we could do that and lower the noise pollution and lower the carbon footprint, while we improve the safety of having these things come to you?”
Parcel delivery by drone is reminiscent of early proposals for mail delivery by parachute, seen here in a 1921 edition of Popular Mechanic.
Agricultural drones
Drones are already being used to help farmers monitor their crops. Companies like PrecisionHawk or senseFly offer aerial imagery capture and analysis of crop growth and health. Drones can cover much larger areas than on the ground monitoring and require minimal upfront costs or investments. With rising population to feed and climate change and soil degradation to combat, drones can be a valuable tool in increasing agricultural yields.
An example of image processing from aerial imagery taken by drone from HoneyComb, one of many companies offering drone services for agriculture
Similarly, Doctors Without Borders is experimenting with drones to rapidly transport patient samples to fight tuberculosis epidemics in parts of Papua New Guinea where road transport is too slow.
Drones are also being used for a variety of archaeological projects. They are a cheap method of capturing images to build 3D models of ruins. “In remote northwestern New Mexico, archaeologists are using drones outfitted with thermal-imaging cameras to track the walls and passages of a 1,000-year-old Chaco Canyon settlement, now buried beneath the dirt. In the Middle East, researchers have employed them to guard against looting.” And in the Yucatan peninsula, drones provided a cost effective solution to flying over dense, remote jungles, and identified previously undiscovered Mayan ruins.
Peruvian archaeologists command a drone to search for architectural ruins. (New York Times)
Of course these are all cool and useful models of non-military uses of drones. But it can get cooler. In the next post, we’ll look at some speculative future uses of drones. Hollywood, get out your pens, or whatever it is you write with these days.
Chris: Oh my drone it’s DRONE WEEK! Wait…what’s drone week?
Recently I was invited to the InfoCamp unConference at Berkeley where among the awesome and inspiring presentations, I sat in on Peter Swigert’s workshop on drones. Since the blog was deep in Oblivion, Pete and I agreed to coauthor a series of posts on this phenomenon, and also to set the record a little more straight for sci-fi fans and authors on the real-world state of drones.
Today, a first post on the scary, scary world of sci-fi drones.
Unmanned (either manual or automated) aerial vehicles, or drones, have become increasingly common in science fiction, likely a reflection of their increasing role in today’s society. While the future of drone technologies and their role in society are yet to be determined, science fiction has been conservative in its speculation. Most depictions of drones tend to suppose an expansion of the current military usage of drone technologies.
Sci-fi: Drones are scary, m’kay
The 2014 remake of Paul Verhoeven’s Robocop has drones that are clearly extensions of modern military drones: wicked Stealth-shaped things with perfect maneuverability for gunning down citizens.
Being welcomed as liberators.
Oblivion takes a similar approach, as drones are fully autonomous, big, scary technospheres used primarily for surveillance, monitoring, and firepower.
Jack facing a drone in bondage.
In Captain America: The Winter Soldier, the drones are gargantuan floating things, each capable of monitoring thousands of square miles at a time, but the concept is the same: drones are a military technology used for surveillance and violence.
The Falcon, for scale.
More recently, Chappie features a very scary military drone who ends up being driven by a psychotic operator against the eponymous, peaceful robot. [Nobody has responded to emails for a screener, so no lovely screen caps for us. -Ed.]
That all you got?
These representations of drones (and many others in the genre) are failures of the creativity of science fiction. While they embody well-founded concerns about drones, they’re monotone, and don’t match the creativity of drone makers right here in the real world.
Hey, I get it. It’s hard to know what drones will mean to future people. Drones combine a whole set of complex technologies that people didn’t know what to do with when they first came out as individual technologies: planes, cameras, GPS, and computers. Sometimes even the military applications aren’t clear. For instance, when the Wright Brothers discussed patenting their approach to the plane in Great Britain and helping the government develop military uses, they were rebuffed, as “Their Lordships are of the opinion that they [airplanes] would not be of any practical use to the Naval Service.”
Additionally, there is an understandable psychological horror at the flying thing that either houses an inhuman machine intelligence or arguably worse, that houses some distant human’s eyes and ears but without their stake in the locale or consequences. Blasted-earth, collateral damage, and horrible mistakes don’t seem to mean as much when the perpetrator can just turn off the monitor and not think about it. So…yes. That military part is pretty scary.
But science fiction films seem particularly confused about how to represent this technology and limited by this military thinking. This was true even before modern military drones were in use; as XKCD notes, The Terminator would have been a much shorter film if it had been developed after Predator drones were around.
From XKCD.com: Our modern military can more effectively abstract the purposes of a drone soldier and design it like a plane, whereas old sci-fi depictions like The Terminator envisioned robotic humans.
But even as science fiction has tackled modern drones, it still builds on military models rather than considering civilian contexts. For instance, the “nanobots” of the 2008 remake of The Day the Earth Stood Still or 2009’s G.I. Joe: Rise of the Cobra suggest what might happen as drone technologies become miniaturized. OK, yes, the military is currently working on this. After all, why send a multimillion dollar Predator that can be shot down when thousands of small drones could more effectively infiltrate enemy territory and conduct operations? That said, this technology doesn’t have to be used in a military context. A micro-sized drone that could identify and poison an enemy’s water supply might be adapted to unclog an artery or kill a cancer cell just as easily.
But even as sci-fi catches up to modern military designs, that’s still only one set of effects for which drones have and might be put to use. In the next post we’ll take a pass at painting the rest of the non-military picture.
The last Scav tech (and the last review of tech in the nerdsourced reviews of Oblivion) is a short one. During the drone assault on the Scav compound, we get a glimpse of the reticle used by the rebel Sykes as he tries to target a weak spot in a drone’s backside.
The reticle has a lot of problems, given Sykes’ task. The data on the periphery is too small to be readable. There are some distracting lines from the augmentation boxes which, if they’re just pointing to static points along the hairline, should be removed. The grid doesn’t seem to serve much purpose. There aren’t good differentiations among the ticks to be able to quickly subitize subtensions. (Read: tell how wide a thing is compared to the tick marks.) (You know, like with a ruler.)
The reticle certainly looks sci-fi, but real-world utility seems low.
The nicest and most surprising thing though is that the bullseye is the right shape and size of the thing he’s targeting. Whatever that circle thing is on the drone (a thermal exhaust port, which seem to be ubiquitously weak in spherical tech) this reticle seems to be custom-shaped to help target it. This may be giving it a lot of credit, but in a bit of apologetics, what if it had a lot of goal awareness, and adjusted the bullseye to match the thing he was targeting? Could it take on a tire shape to disable a car? Or a patella shape to help incapacitate a human attacker? That would be a very useful reticle feature.
As Jack searches early in the film for Drone 172, he parks his bike next to a sinkhole in the desert and cautiously peers into it. As he does so, he is being observed from afar by a sinister looking Scav through a set of asymmetrical…well, it’s not exactly right to call them binoculars.
They look kind of binocular, but that term technically refers to a machine that displays two slighty-offset images shown independently to each eye such that the user perceives stereopsis, or a single field in 3D. But a quick shot from the Scav’s perspective shows that this is not what is shown at all. Continue reading →