Everyone has played the piano on a table or part of the body. But it probably didn’t sound like it did with any of them. Thanks to an invention by ETH Zurich, this is now possible: using a wrist sensor that converts finger movements into computer commands – mobile and cheap.

The simulation of environments, coupled with technologies such as virtual reality glasses, offers almost unlimited possibilities. However, VR systems are not used much in everyday life. “The technologies have not yet established themselves in productive activities, such as classic office work,” observed Christian Holz, professor at the Institute for Intelligent Interactive Systems at ETH Zurich.

There is also great potential here: content is no longer tied to a screen, users can act three-dimensionally, highly flexibly and intuitively with their hands. The central problem lies in the interaction between people and technology, says Holz.

So far, the devices have been operated with controllers or hands-free in the air by filming the movements. Working standing up with your hands raised is tiring. “A normal work process, i.e. the interaction with the system over several hours, is unthinkable”. And with virtual keyboards, the finger movements are too small to be implemented precisely by cameras.

For both problems, passive surfaces offer a sensible alternative, according to Holz and his team. This can be the classic table top, but also a wall or your own body. The sensor principle called “TapID”, embedded in a cheap rubber bracelet, can resolve tiny differences in the vibration profile on the wrist that are characteristic of every finger. The movement of each individual finger manifests itself differently on the wrist, and the “TapID” device makes use of these differences.

A specially developed machine learning pipeline processes the measured data in real time. In combination with the camera system of VR glasses, which records the position of the hands, an extremely precise input is possible. The researchers demonstrate this in a number of applications that they programmed for their development, such as a virtual keyboard and a piano.

In a technical evaluation with 18 participants, the developers were able to show that the principle not only works reliably with the specially designed electronics in the bracelet, but also with fitness bracelets and smartwatches from established providers that are equipped with inertia sensors.

In addition, the bracelet is cheap to manufacture and easy to take anywhere, says computer science professor Holz. He sees the future of virtual reality in being able to work together regardless of physical position – not restricted by hardware, but as if one were in the same room. (apa / sda)