Acoustic-based Upper Facial Action Recognition for Smart Eyewear

Xie，Wentao1; Zhang，Qian2; Zhang，Jin3

2021-06-01

10.1145/3448105

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies   影响因子和分区

2474-9567

Smart eyewear (e.g., AR glasses) is considered to be the next big breakthrough for wearable devices. The interaction of state-of-the-art smart eyewear mostly relies on the touchpad which is obtrusive and not user-friendly. In this work, we propose a novel acoustic-based upper facial action (UFA) recognition system that serves as a hands-free interaction mechanism for smart eyewear. The proposed system is a glass-mounted acoustic sensing system with several pairs of commercial speakers and microphones to sense UFAs. There are two main challenges in designing the system. The first challenge is that the system is in a severe multipath environment and the received signal could have large attenuation due to the frequency-selective fading which will degrade the system's performance. To overcome this challenge, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel state information (CSI) estimation scheme that is able to measure the phase changes caused by a facial action while mitigating the frequency-selective fading. The second challenge is that because the skin deformation caused by a facial action is tiny, the received signal has very small variations. Thus, it is hard to derive useful information directly from the received signal. To resolve this challenge, we apply a time-frequency analysis to derive the time-frequency domain signal from the CSI. We show that the derived time-frequency domain signal contains distinct patterns for different UFAs. Furthermore, we design a Convolutional Neural Network (CNN) to extract high-level features from the time-frequency patterns and classify the features into six UFAs, namely, cheek-raiser, brow-raiser, brow-lower, wink, blink and neutral. We evaluate the performance of our system through experiments on data collected from 26 subjects. The experimental result shows that our system can recognize the six UFAs with an average F1-score of 0.92.

acoustic sensing eyewear facial actions OFDM wearables

EI

英语

第一

20212710577068

Channel state information ; Convolutional neural networks ; Face recognition ; Frequency domain analysis ; Frequency estimation ; Glass ; Orthogonal frequency division multiplexing

Radio Systems and Equipment:716.3 ; Glass:812.3 ; Mathematical Transformations:921.3

2-s2.0-85108895759

Scopus

1.Department of Computer Science and Engineering,The Hong Kong University of Science and Technology,Southern University of Science and Technology,Shenzhen,China
2.Department of Computer Science and Engineering,The Hong Kong University of Science and Technology,Hong Kong,Hong Kong
3.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,China

Xie，Wentao,Zhang，Qian,Zhang，Jin. Acoustic-based Upper Facial Action Recognition for Smart Eyewear[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies,2021,5(2).

Xie，Wentao,Zhang，Qian,&Zhang，Jin.(2021).Acoustic-based Upper Facial Action Recognition for Smart Eyewear.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies,5(2).

Xie，Wentao,et al."Acoustic-based Upper Facial Action Recognition for Smart Eyewear".Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5.2(2021).