Accelerometer-Based Hand Gesture Recognition by Neural Network and Similarity Matching.docx

《Accelerometer-Based Hand Gesture Recognition by Neural Network and Similarity Matching.docx》由会员分享，可在线阅读，更多相关《Accelerometer-Based Hand Gesture Recognition by Neural Network and Similarity Matching.docx（9页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、IEEE SENSORS JOURNAL, VOL. 16, NO. 11, JUNE 1, 2016 4537 G Accelerometer-Based Hand Gesture Recognition by Neural Network and Similarity Matching Renqiang Xie and Juncheng Cao Abstract In this paper, we present an accelerometer-based pen-type sensing device and a user-independent hand gesture recogn

2、ition algorithm. Users can hold the device to perform hand gestures with their preferred handheld styles. Gestures in our system are divided into two types: the basic gesture and the complex gesture, which can be represented as a basic gesture sequence. A dictionary of 24 gestures, including 8 basic

3、 gestures and 16 complex gestures, is defined. An effective segmentation algorithm is developed to identify individual basic gesture motion intervals automatically. Through segmentation, each complex gesture is segmented into several basic gestures. Based on the kinematics characteristics of the bas

4、ic gesture, 25 features are extracted to train the feedforward neural network model. For basic gesture recognition, the input gestures are classified directly by the feedforward neural network classifier. Nevertheless, the input complex gestures go through an additional similarity matching procedure

5、 to identify the most similar sequences. The proposed recognition algorithm achieves almost perfect user- dependent and user-independent recognition accuracies for both basic and complex gestures. Experimental results based on 5 subjects, totaling 1600 trajectories, have successfully validated the e

6、ffectiveness of the feedforward neural network and similarity matching-based gesture recognition algorithm. Index Terms Accelerometer, gesture recognition, gesture segmentation, feedforward neural network, similarity matching. I. INTRODUCTION ESTURE recognition refers to the process of understanding

7、 and classifying meaningful movements of a humans fingers, hands, arms or head 1. Hand gesture as a natural, intuitive, and convenient way of human-computer interaction (HCI) will greatly ease the interaction process. For instance, in 2, a hand gesture recognition based motion control system of inte

8、lligent wheelchair is developed for those with physical accessibility problem; five gestures are employed to separately control the motion of the wheelchair: Manuscript received January 15, 2016; revised March 23, 2016; accepted March 23, 2016. Date of publication March 25, 2016; date of current ver

9、sion April 26, 2016. This work was supported in part by the 863 Program of China under Project 2011AA010205, in part by the National Natural Science Foundation of China under Grant 61131006, and in part by the 973 Program of China under Grant 2014CB339803. The associate editor coordinating the revie

10、w of this paper and approving it for publication was Prof. Octavian Postolache. R. Xie is with the Key Laboratory of Terahertz Solid-State Technology, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200050, China, and also with the School of Inform

11、ation Science and Technology, ShanghaiTech University, Shanghai 201210, China (e-mail: ). J. C. Cao is with the Key Laboratory of Terahertz Solid-State Tech- nology, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200050, China (e-mail: jccaomail.

12、). Digital Object Identifier 10.1109/JSEN.2016.2546942 left turn, right turn, forward, backward, and stop. Other proposed applications of hand gesture recognition include robot-assisted living 3, automatic user state recognition for low-cost television control system 4, and smart ring 5. Conventiona

13、l computer vision-based hand gesture recognition can track and recognize gestures effectively without any contact to the user 6, 7. However, vision- based techniques may be affected by lighting conditions, which will limit the application scenarios, particularly in mobile environment. With the rapid

14、 development of sensor technology, triaxial accelerometers are being increasingly embedded into consumer electronic products. A significant advantage of accelerometer-based sensing devices is that they can be operated without any external reference or limitation in working conditions 8. Hand gesture

15、 recognition is relatively complicated since different persons have different speeds and styles to perform gestures. Thus, some researchers have tried to combine data from a triaxial accelerometer with data from electromyography (EMG) sensors 9, 10 or vision sensors 11, 12 in order to improve the sy

16、stems performance and robustness. However, multi-sensor fusion increases additional cost as well as computational burden. Concerning the recognition methodologies, Hidden Markov Model (HMM) and Dynamic Time Warping (DTW) as two important approaches are widely used to recognize hand gestures effectiv

17、ely 2, 3, 9, 13, 14. Other proposed methods include Probabilistic Neural Network (PNN) 8, Most Probable Longest Common Subsequence (MPLCS) 15, Sign Sequence and Template Matching (SSTM) 16, and Stochastic Linear Formal Grammar (SLFG) 17. Generally, the subjects from which the trajectories are collec

18、ted to con- struct the classifier are not consistent with the end users of the system. To develop a user-independent algorithm, Akl et al. 1 proposed an accelerometer-based gesture recognition system, which employed Dynamic Time Warping and Affinity Propa- gation (DTW & AP) algorithms to create exem

19、plars for each gesture during the training stage. A database of 3780 traces was created for a dictionary of 18 gestures. The system achieves accuracies of 99.81% and 94.60% for user-dependent and user-independent recognitions for the 18 gestures, respectively. In this paper, an accelerometer-based p

20、en-type sensing device as well as a Feedforward Neural Network and Similarity Matching (FNN & SM) based hand gesture recog- nition algorithm are presented. The work of this paper is built upon a preliminary version of our gesture recognition system 5. The accelerations generated by hand movements ar

21、e collected and transmitted to a personal computer (PC) via 1558-1748 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http:/www.ieee.org/publications_standards/publications/rights/index.html for more information. 4538 IEEE SENSORS JOURNAL, VOL. 16

22、, NO. 11, JUNE 1, 2016 Fig. 2. Schematic diagram of the pen-type sensing device. Fig. 1. Sensing device and its coordinate system. a USB cable. Users can hold the device to perform hand gestures with their preferred handheld styles in free space. Gestures in our system are divided into two types: th

23、e basic gesture and the complex gesture which can be represented as a basic gesture sequence. The gesture recognition algorithm is composed of data acquisition and signal preprocessing, gesture segmentation, feature extraction, classifier construction, basic gesture encoding, and similarity matching

24、. For basic gesture recognition, the input gestures are classified directly by the Fig. 3. Trajectories of eight basic gestures. FNN classifier. Nevertheless, the input complex gestures go through an additional similarity matching procedure to identify the most similar sequences. The main contributi

25、ons of this paper include the following: 1) a segmentation scheme is proposed to identify the starting and end points of each basic gesture automatically; through segmentation, each complex gesture is segmented into several basic gesture motion inter- vals; 2) based on the kinematics characteristics

26、 of the basic gesture, 25 effective features are extracted; 3) the FNN and SM algorithms are successfully combined; the FNN classifier provides good recognition accuracy, while the SM approach enhances the extendibility of the system; 4) the contrast test for user-dependent and user-independent clas

27、sifications is conducted to validate the user dependency of the proposed system. The remainder of this paper is organized as follows. We first introduce the accelerometer-based pen-type sensing device in Section II. The FNN and SM based hand gesture recognition algorithm is elaborated detailedly in

28、Section III. In Section IV, experimental results are presented and discussed to validate the effectiveness of the proposed approach. Finally, conclusions are given in the last section. II. SENSING SYSTEM OVERVIEW The pen-type gesture capturing device, which is shown in Fig. 1, consists of a triaxial

29、 MEMS (Micro-electromechanical Systems) accelerometer (MMA9551L) and a microcon- troller (IAP15W4K58S4). The MMA9551L possesses a user- selectable acceleration full scale of 2 g, 4 g, and 8 g, and its supply voltage range is 1.71 V 1.89 V. The IAP15W4K58S4 is a new generation of 8051 MCU with high s

30、peed, wide voltage range (2.5 V 5.5 V), low power consumption, and has a fully compatible instruction set with traditional 8051 series microcontroller. In our system, the supply voltages of MMA9551L and IAP15W4K58S4 are set to 1.8 V and 3.3 V, respectively. Due to mismatching of levels, a voltage-le

31、vel translator (PCA9306) is added between them. The schematic diagram of the pen-type sensing device is shown in Fig. 2. The accelerometer measures the analog acceleration signals generated by a users hand movements and converts the Fig. 4. Trajectories of sixteen complex gestures and their correspo

32、nding numbers. signals to digital ones via the internal 16-bits A/D converter. The microcontroller collects the digital acceleration signals from the accelerometer through IIC interface and transmits the signals to a PC via a USB cable for further signal processing and analysis. The accelerometers s

33、ensitivity is set from 8 g to +8 g in this study, and the output signals of the accelerometer are sampled at 100 Hz. Note that all signal processing procedures presented in Section III are performed on the PC. The size of the pen-type circuit board is 14 cm 2.4 cm 1.5 cm and the corresponding coordi

34、nate system of the accelerometer is illustrated in Fig. 1. III. HAND GESTURE RECOGNITION The gesture recognition algorithm is composed of data acquisition and signal preprocessing, gesture segmentation, feature extraction, classifier construction, basic gesture encod- ing, and similarity matching. I

35、n this paper, a dictionary of 24 gestures including 8 basic gestures (see Fig. 3) and 16 complex gestures (see Fig. 4) is defined. Since the basic gesture recognition procedures are part of those of complex gesture recognition, we will focus on the whole procedures of the latter. The acceleration si

36、gnals of hand movements are measured by a triaxial accelerometer and then preprocessed by a moving average filter. A segmentation algorithm is developed to identify the starting and end points of each basic gesture automatically. After feature extraction, the basic gesture samples are used to train

37、the FNN model which subsequently is utilized to classify the basic gesture sequences. The recognized basic gesture sequence is then encoded with Johnson codes. Finally, the complex gesture is recognized by comparing the similarity between the predicted basic gesture XIE AND CAO: ACCELEROMETER-BASED

38、HAND GESTURE RECOGNITION 4539 + which is expressed as 1 M an = 2M + 1 m=M as n + m (1) where as n is the acceleration signal without gravitational acceleration, and an is the filtered acceleration signal. In this study, we set M = 5, i.e., the number of points in the filter is 11. Fig. 5. Block diag

39、ram of the proposed hand gesture recognition algorithm. sequence and the standard template sequences. Note that for basic gesture recognition, it does not include the encoding and matching procedures, and is classified directly by the trained FNN classifier. The block diagram of the proposed hand ge

40、sture recognition algorithm is shown in Fig. 5 and the detailed procedures of the algorithm are introduced as follows. A. Gesture Introduction C. Gesture Segmentation Gesture segmentation aims to identify the starting and end points of each motion from the preprocessed accelerations. Reliable and ac

41、curate segmentation is essential to gesture recognition. Each complex gesture is segmented into two or more basic gesture motion intervals by the segmentation program. In 1 and 8, a gesture trace is directly seg- mented by pressing and releasing a button. However, this is not always consistent with

42、the real situation, i.e., the starting and end points of a motion are not the time pressing and releasing the button respectively. Here we propose a novel segmentation scheme. Given the preprocessed accel- eration data sequence AS = a1, a2, , aL, where an = (ax n, ay n, az n) is a three-dimensional

43、vector. dn is defined as the Euclidean distance between an and an 1. The acceleration is relatively stable when there is no hand movement. In contrast, it varies dramatically when in hand motion state. This means dn in motion state is much higher than in status of no movement, hence it can be used t

44、o segment a gesture motion. In order to avoid the disturbances of the signal, a moving average filter is applied, N The 8 basic gestures including right (R), left (L), up (U), down (D), upper right (UR), lower left (LL), upper left (UL), 1 J n = 2N 1 m=N dn + m (2) and lower right (LR) are shown in

45、Fig. 3, while the 16 complex gestures and their corresponding numbers (116) are shown in Fig. 4. The basic gesture movements can be extended by users as they can build their own profiles of sequences. For instance, the gesture square (gesture number is 16) can be split into up, right, down, and left

46、. Hence a basic gesture sequence of U-R-D-L is generated. B. Data Acquisition and Signal Preprocessing The raw acceleration signals of hand movements are gen- erated by the triaxial accelerometer and collected by the microcontroller. In order to reduce the influence of unintended hand motions, a but

47、ton is employed to trigger the sampling. Specifically, the button should be pressed before performing a gesture and released after the gesture is completed. The measured signals are always contaminated by the disturbances of the sensor as well as users unconscious trembles. To settle this problem, f

48、irst, the gravitational acceleration is removed by subtracting the mean value of sampled accelerations from each data point to obtain accelerations generated by hand movement. The next step of the signal preprocessing is to reduce high frequency noise by using a moving average filter where J n is the filtered Euclidean distance and N = 5. The preprocessed accelerations and J of gesture right, 6, 14, and 16 are shown in Fig. 6. The accelerations are mainly on x- and y-axis. Note that each basic gesture corresponds to a main peak (marked with a circle) of J