Landon Cox | Duke University

Moving to MSR

After 13 years at Duke, I moved to Microsoft Research to join the Mobility and Networking Group in the summer of 2018.

Privacy markers at MobiSys

Ashwin Machanavajjhala and I have been working with our students to find better ways of controlling the information that mobile apps access through a device's camera. Existing operating systems provide only coarse-grained control, i.e., an app has permission to access all camera data or none, but there is no way to allow partial access. For example, there is no good way to video chat with someone while keeping them from reading the whiteboard behind you. Today, either you give the video-chat app full access to the camera or none.

Our recent MobiSys paper provides finer grained control over visual information through privacy markers. A privacy marker allows users to tell the OS exactly which two-dimensional surfaces and three-dimensional objects in a video stream are important and should be treated with care. Users mark two-dimensional areas by drawing a special shape, e.g., on a piece of paper, on a whiteboard, or within a projected presentation. Users mark three-dimensional objects through a trusted UI on their device.

The OS can treat privacy markers in two ways: as secrets that should be blocked from a video stream, or as public areas so that everything except the marked areas are blocked. We initially treated marked areas as secret, but found that the computer-vision algorithms needed to detect the markers were not robust enough to provide strong security. Too often, lighting, motion blur, and other factors caused computer vision to fail and reveal secret information. Instead, we treated marked regions as public; the camera stream is blocked by default, and the system reveals only the marked regions that it detects. This sacrifices utility in favor of stronger privacy, but we found that this trade-of was appropriate for many (if not most) smartphone apps.

We built two systems around the idea of privacy markers, PrivateEye and WaveOff, and integrated both into the Android camera sub-system. A lot of engineering effort went into supporting realtime marker detection, which is critical for many mobile apps. Our initial prototype supported an unusable 4 FPS, but by the end the grad students had gotten throughput ramped up to a perfectly reasonable 23 FPS (on a fairly old smartphone).

Looking to the future, while the focus of this work was smartphone apps like video chat, document scanning, and QR-code scanning, the issue of fine-grained control of visual information is of much broader importance. Perception-driven systems like robots seem increasingly likely to enter our homes and offices, and it will be critical to ensure that these systems access only as much visual information as they need to do their jobs. We think that privacy markers can be an important part of making these emerging systems privacy sensitive.