New Technology Could Revolutionise Smartphone Use

Estimated reading time: 2 minutes

An innovative technology, created by researchers at the University of St Andrews, can allow your smartphone to carry out a range of tasks just by recognising the surface it is sitting on.

SpeCam is a program which would enable existing phones to utilize their camera function to recognise different materials they have been exposed to. This would link up with a previously created database so the phone could “understand” what type of surface it has been placed on.

This offers a wide range of applications from allowing your phone to tell you where it is when you can’t find it, to programming it to take different surfaces as a trigger for action. For example if the phone rings and you place it face down on your laptop, it could send a message saying “sorry I am in a meeting” or if put inside your pocket it could suggest the caller try a different colleague.

The program would also allow the way the phone is held to perform tasks, such as changing the volume on the television, or playing certain music, such as soft jazz if placed on the sofa or dance music if on the coffee table.

‌As the required hardware already exists on modern smartphones, the SpeCam system could be enabled through a software update or application download.

The research will be presented at the prestigious 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, ACM SIGCHI MobileHCI 2017, on Wednesday 6 September in Vienna, Austria. The paper ‘SpeCam: Sensing Surface Color and Material with the Front-Facing Camera of a Mobile Device’ has received an honourable mention award at MobileHCI, one of only five such papers in the entire program.

Lead researcher Professor Aaron Quigley, Chair of Human Computer Interaction at the University, said: “This is an example of what we call Discreet Computing or discreet interaction, where subtle and inconspicuous user actions can result in entirely new forms of interaction. By training and hence recognising materials all the surfaces around us become a canvas for our imagination.”

SpeCam cleverly uses the screen/display on a smartphone as a multi-spectral light source and the front facing camera to capture the reflection of the material it has been placed face-down upon.

a) The phone is facing down while the screen flashes different colors and the camera captures images of the reflected light.
b) The UI of the phone, showing the main light source and the captured images.
c) The UI of the classification server, showing the received images, the classification results on top and the extracted features (color histogram, gradient, etc).
d) 3D printed cases with different height for the pilot study.

The program allows the way the screen rapidly cycles through different colours to illuminate the object to detect the material and infer the location or placement of the device.

Hui Yeo, whose PhD research led to this research, added: “The goal is to explore novel forms of interaction for one-handed use which can leverage what people already do with devices, without adding new hardware to the device.”