Pentelligence: Handwriting Recognition with Motion and Audio Data from Pens

Pentelligence is a novel interaction device that does not require special paper or additional hardware (except for a computer) to recognize handwriting. This pen can achieve recognition rates of over 98% for digits on individual users without writing instructions. Its main strength lies in the use of deep neural networks that learn the individual writing styles of the users. By combining motion and audio data from inside the pen, the strengths of the individual sensors complement each other.
On this page we offer a short introduction to our project and give a brief overview of the main features of Pentelligence. For further information, please refer to our paper: Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition

Developed by the Human-Computer Interaction group at the University of Hannover.


Improved digital note-taking by combining audio and motion.

Click here to watch preview.


Painting on the back of the hand with magnetic tracking and motion sensing.

Click here to watch preview.

Click to watch video

Vision

Note-taking in the digital age?

Click here to learn more about the vision of Pentelligence.

Hardware

Take a look inside and explore the concept!

Click here to see how the heart of our digital pen is designed.

Recognition

Audio & motion data from pens?

Learn more about recognizing handwritten digits.

Note-taking in the digital age

For thousands of years humans wrote down their ideas in various ways. Even today, the pen is an indispensable part of our lives. Although even tablets offer precise input options, many people still prefer the pen and paper.

The vision of Pentelligence is to bridge the analog and digital worlds with a device that doesn't differ from the appearance of ordinary pens. In our paper at the CHI 2018 we have shown the concept and the recognition of digits, which is particularly important because, in contrast to words, in many cases no subsequent autocorrection can be performed.

The project will continue to pursue the vision of a digital pen that does not require special paper, any other additional hardware or even cameras to recognize handwriting.

A look inside

From the inside, Pentelligence relies on small, inexpensive and easily available hardware components. With a length of 12cm and a diameter of 12mm, the prototype almost reaches the dimensions of a conventional ballpoint pen in the first phase.

A microcontroller with an easy-to-program Arduino bootloader records the measured values of the microphone and the inertial measuring unit together with a specially self-developed binary writing pressure sensor and sends them to the computer via USB.

In future releases a wireless version will be developed, which will allow an even more realistic look and feel compared to ordinary pens.

Deep learning

Handwritten digits are as unique as their writers. For this reason, modern methods of deep learning are necessary to customize Pentelligence to the individual writing style.

The envelopes of the writing audios, fed into the neural networks, eliminate surface-related features. Downsampled and averaged motion data together with write pressure information capture individual strokes and reconstruct the writing trajectories within the classifiers.

By subsequently combining the results of the audio and motion classifiers, the complementary features can be merged and recognition rates of over 98% can be achieved.

2020
Watch my Painting: The Back of the Hand as a Drawing Space for Smartwatches Maximilian Schrapel, Florian Herzog, Steffen Ryll, Michael Rohs Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems
Poster
        
Regulating Access in Office Environments with Digital Pens Maximilian Schrapel Proceedings of the 1st International Workshop on Authentication Beyond Desktops and Smartphones: Novel Approaches for Smart Devices and Environments
Workshop Paper
     
2018
Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition Maximilian Schrapel, Max-Ludwig Stadler, Michael Rohs Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Full Paper
        
Digital pens emit ink on paper and digitize handwriting. The range of the pen is typically limited to a special writing surface on which the pen's tip is tracked. We present Pentelligence, a pen for handwritten digit recognition that operates on regular paper and does not require a separate tracking device. It senses the pen tip's motions and sound emissions when stroking. Pen motions and writing sounds exhibit complementary properties. Combining both types of sensor data substantially improves the recognition rate. Hilbert envelopes of the writing sounds and mean-filtered motion data are fed to neural networks for majority voting. The results on a dataset of 9408 handwritten digits taken from 26 individuals show that motion+sound outperforms single-sensor approaches at an accuracy of 78.4% for 10 test users. Retraining the networks for a single writer on a dataset of 2120 samples increased the precision to 100% for single handwritten digits at an overall accuracy of 98.3%.

Pentelligence is developed by the Human-Computer Interaction Group at the University of Hannover. For additional details, read our paper (available on our homepage or in the ACM digital library). For inquiries, please contact: maximilian.schrapel@hci.uni-hannover.de

© Copyright Human-Computer Interaction Group, University of Hannover 2019