Eye Phone

As smartphones evolve researchers are studying new techniques to ease the human-mobile interaction. We propose EyePhone, a novel “hand-free” interfacing system capable of driving mobile applications/functions using only the user’s eyes movement and actions (e.g., wink). EyePhone tracks the user’s eye movement across the phone’s display using the camera mounted on the front of the phone; more specifically, machine learning algorithms are used to: i) track the eye and infer its position on the mobile phone display as a user views a particular application; and ii) detect eye blinks that emulate mouse clicks to activate the target application under view. We present a prototype implementation of EyePhone on a Nokia N810, which is capable of tracking the position of the eye on the display, mapping this positions to an application that is activated by a wink. At no time does the user have to physically touch the phone display.

Human-Computer Interaction (HCI) researchers and phone vendors are continuously searching for new approaches to reduce the effort users exert when accessing applications on limited form factor devices such as mobile phones. The most significant innovation of the past few years is the adoption of touchscreen technology introduced with the Apple iPhone [1] and recently followed by all the other major vendors, such as Nokia [2] and HTC [3]. The touchscreen has changed the way people interact with their mobile phones because it provides an intuitive way to perform actions using the movement of one or more fingers on the display (e.g., pinching a photo to zoom in and out, or panning to move a map).

Human-Phone Interaction

Human-Phone Interaction represents an extension of the field of HCI since HPI presents new challenges that need to be addressed specifically driven by issues of mobility, the form factor of the phone, and its resource limitations (e.g., energy and computation). More specifically, the distinguishing factors of the mobile phone environment are mobility and the lack of sophisticated hardware support, i.e., specialized headsets, overhead cameras, and dedicated sensors, that are often required to realize HCI applications. In what follows, we discuss these issues. Mobility Challenges. One of the immediate products of mobility is that a mobile phone is moved around through unpredicted context, i.e., situations and scenarios that are hard to see or predict during the design phase of a HPI application.

A mobile phone is subject to uncontrolled movement, i.e., people interact with their mobile phones while stationary, on the move, etc. It is almost impossible to predict how and where people are going to use their mobile phones. A HPI application should be able to operate reliably in any encountered condition. Consider the following examples: two HPI applications, one using the accelerometer, the other relying on the phone’s camera. Imagine exploiting the accelerometer to infer some simple gestures a person can perform with the phone in their hands, e.g., shake the phone to initiate a phone call, or tap the phone to reject a phone call .

What is challenging is being able to distinguish between the gesture itself and any other action the person might be performing. For example, if a person is running or if a user tosses their phone down on a sofa, a sudden shake of the phone could produce signatures that could be easily confused with a gesture. There are many examples where a classifier could be easily confused. In response, erroneous actions could be triggered on the phone. Similarly, if the phone’s camera is used to infer a user action [5][9], it becomes important to make the inference algorithm operating on the video captured by the camera robust against lighting conditions, which can vary from place to place. In addition, video frames blur due to the phone movement.

Because HPI application developers cannot assume any optimal operating conditions (i.e., users operating in some idealized manner) before detecting gestures in this example, (e.g., requiring a user to stop walking or running before initiating a phone call by a shaking movement), then the effects of mobility must be taken into account in order for the HPI application to be reliable and scalable. Hardware Challenges. As opposed to HCI applications, any HPI implementation should not rely on any external hardware. Asking people to carry or wear additional hardware in order to use their phone might reduce the penetration of the technology.

Moreover, state-of-the art HCI hardware, such as glass mounted cameras, or dedicated helmets are not yet small enough to be conformably worn for long periods of time by people. Any HPI application should rely as much as possible on just the phone’s on-board sensors. Although modern smartphones are becoming more computationally capable , they are still limited when running complex machine learning algorithms [14]. HPI solutions should adopt lightweight machine learning techniques to run properly and energy efficiently on mobile phones.

Eyephone Design

One question we address in this paper is how useful is a cheap, ubiquitous sensor, such as the camera, in building HPI applications. We develop eye tracking and blink detection mechanisms based algorithms [13, 17] originally designed for desktop machines using USB cameras. We show the limitations of an off-the-shelf HCI technique [13] when used to realize a HPI application on a resource limited mobile device such as the Nokia N810.

The EyePhone algorithmic design breaks down into the following pipeline phases:

1) an eye detection phase;

2) an open eye template creation phase;

3) an eye tracking phase;

4) a blink detection phase. In what follows, we discuss each of the phases in turn. Eye Detection.

By applying a motion analysis technique which operates on consecutive frames, this phase consists on finding the contour of the eyes. The eye pair is identified by the left and right eye contours. While the original algorithm identifies the eye pair with almost no error when running on a desktop computer with a fixed camera (see the left image in Figure 1), we obtain errors when the algorithm is implemented on the phone due to the quality of the N810 camera compared to the one on the desktop and Figure 1: Left figure: example of eye contour pair returned by the original algorithm running on a desktop with a USB camera. The two white clusters identify the eye pair. Right figure: example of number of contours returned by EyePhone on the Nokia N810.

The smaller dots are erroneously interpreted as eye contours. the unavoidable movement of the phone while in a person’s hand (refer to the right image in Figure 1). Based on these experimental observations, we modify the original algorithm by: i) reducing the image resolution, which according to the authors in reduces the eye detection error rate, and ii) adding two more criteria to the original heuristics that filter out the false eye contours. In particular, we filter out all the contours for which their width and height in pixels are such that widthmin ≤ width ≤ widthmax and heightmin ≤ height ≤ heightmax.

The widthmin, widthmax, heightmin, and heightmax thresholds, which identify the possible sizes for a true eye contour, are determined under various experimental conditions (e.g., bright, dark, moving, not moving) and with different people. This design approach boosts the eye tracking accuracy considerably

Applications:

EyeMenu

An example of an EyePhone application is EyeMenu as shown in Figure 4. EyeMenu is a way to shortcut the access to some of the phone’s functions. The set of applications in the menu can be customized by the user. The idea is the following: the position of a person’s eye is mapped to one of the nine buttons. 

A button is highlighted when EyePhone detects the eye in the position mapped to the button. If a user blinks their eye, the application associated with the button is lunched. Driving the mobile phone user interface with the eyes can be used as a way to facilitate the interaction with mobile phones or in support of people with disabilities.

Car Driver Safety

EyePhone could also be used to detect drivers drowsiness and distraction in cars. While car manufactures are developing technology to improve drivers safety by detecting drowsiness and distraction using dedicated sensors and cameras [22], EyePhone could be readily usable for the same purpose even on low-end cars by just clipping the phone on the car dashboard.

Copyright By

Emiliano Miluzzo, Tianyu Wang, Andrew T. Campbell, Computer Science Department, Dartmouth College,Hanover, NH, USA

MobiHeld 2010, August 30, 2010, New Delhi, India.