Landon Cox | Duke University

Privacy markers at MobiSys

Ashwin Machanavajjhala and I have been working with our students to find better ways of controlling the information that mobile apps access through a device's camera. Existing operating systems provide only coarse-grained control, i.e., an app has permission to access all camera data or none, but there is no way to allow partial access. For example, there is no good way to video chat with someone while keeping them from reading the whiteboard behind you. Today, either you give the video-chat app full access to the camera or none.

Our recent MobiSys paper provides finer grained control over visual information through privacy markers. A privacy marker allows users to tell the OS exactly which two-dimensional surfaces and three-dimensional objects in a video stream are important and should be treated with care. Users mark two-dimensional areas by drawing a special shape, e.g., on a piece of paper, on a whiteboard, or within a projected presentation. Users mark three-dimensional objects through a trusted UI on their device.

The OS can treat privacy markers in two ways: as secrets that should be blocked from a video stream, or as public areas so that everything except the marked areas are blocked. We initially treated marked areas as secret, but found that the computer-vision algorithms needed to detect the markers were not robust enough to provide strong security. Too often, lighting, motion blur, and other factors caused computer vision to fail and reveal secret information. Instead, we treated marked regions as public; the camera stream is blocked by default, and the system reveals only the marked regions that it detects. This sacrifices utility in favor of stronger privacy, but we found that this trade-of was appropriate for many (if not most) smartphone apps.

We built two systems around the idea of privacy markers, PrivateEye and WaveOff, and integrated both into the Android camera sub-system. A lot of engineering effort went into supporting realtime marker detection, which is critical for many mobile apps. Our initial prototype supported an unusable 4 FPS, but by the end the grad students had gotten throughput ramped up to a perfectly reasonable 23 FPS (on a fairly old smartphone).

Looking to the future, while the focus of this work was smartphone apps like video chat, document scanning, and QR-code scanning, the issue of fine-grained control of visual information is of much broader importance. Perception-driven systems like robots seem increasingly likely to enter our homes and offices, and it will be critical to ensure that these systems access only as much visual information as they need to do their jobs. We think that privacy markers can be an important part of making these emerging systems privacy sensitive.

Kahawai at MobiSys

It was great to see former student Eduardo Cuervo present our paper on Kahawai at MobiSys ’15 last week.

Kahawai is a new approach to cloud gaming that allows a computer with an under-powered GPU, such as a mobile device or an older-generation gaming console, to render the same visual output as a computer with a top-of-the-line GPU. There are a number of commercial cloud-gaming systems, such as Ubitus and PlayStation Now, that aim to do the same thing, and these services take a thin-client approach to cloud gaming. That is, the game executes on a server equipped with a powerful GPU, and the game output is encoded as compressed video and streamed to clients. The problem with a thin-client approach is that the bandwidth requirements are substantial, with services typically requiring a minimum of 5 Mbps between the server and each client.

Eduardo had two insights into this bandwidth problem. First, even though a computer might not have the latest 300 W GPU, it probably still has a perfectly capable GPU. For example, smartphones have increasingly powerful GPUs, even if they will never render the same level of detail at the same rate as a dedicated gaming rig. Thin clients leave these perfectly capable GPUs idle since the remote server is completely responsible for generating the game’s output. A better approach might be to use the client’s GPU to help render the game’s graphics and reduce the amount of data the server has to stream. Eduardo’s second insight was that games naturally support multiple levels of visual fidelity. Most games allow a player to adjust their settings to match the power of their hardware, so that players with less powerful hardware can play the game with lower-fidelity graphics.

Based on these observations, we developed the idea of collaborative rendering, in which the client generates a low-fidelity version of the game’s graphics using its local GPU, and the server sends only the missing details from its high-fidelity version of the output (rather than the entire graphical output). Kahawai supports two implementations of collaborative rendering, one in which the client generates low-detail graphics at a high rate (a technique called delta encoding) and one in which the client generates high-detail graphics at a low rate (a technique called client-side i-frame rendering).

Results from experiments with our Kahawai prototype are quite promising. For Doom 3, Kahawai uses 1/6th of the bandwidth of a thin client, and a user study with 50 Microsoft gamers demonstrated that the gaming experience of Kahawai and a thin client are essentially equivalent. The upshot is that Kahawai gives players the same gaming experience as a conventional cloud-gaming service without burning through their cellular data or clogging their home networks, and at the same time Kahawai significantly cuts the bandwidth costs for cloud-gaming service providers.

Kahawai took several years to complete and many people contributed. Eduardo started the project while interning at Microsoft Research with Alec Wolman, Stefan Saroiu, and others, made significant progress after coming back to Duke, and tied up all the loose ends after joining MSR full-time.