Eye Blinks in French Sign Language Definition of eye blink types and automatic detection of eye blinks using computer vision, rule-based and machine learning-based methods.

Abstract

This study aims at defining the different types of eye blinks in French Sign Language as well as preparing a potential automatic eye blink types classification in creating a method that allows the detection of blinks in a systematic and reliable way. In this thesis, we learn about the research done on non-manual markers in sign languages. We focus on eye blinks and mention studies which are interested in blinks in sign languages and then studies which have contributed to the creation of automatic eye blink detection methods in various fields, ranging from medicine to automatic engineering. Following, we present the different phases of annotation of our data. We define the different types of blinks, both linguistic and non-linguistic, that we have found in our dataset. We then go on presenting the methods that we use to detect blinks automatically. Finally, we report our results, proposing a proof of concept for automatic eye blink detection that combines a CNN regression model and logic rules. We analyze ours results and we show that the Eye Aspect Ratio calculation used in combination with a cascade classifier in most methods for eye blink detection though robust (as the EAR calculation relies on the reliability of the landmarks detector) might be outperformed by the combination of a CNN algorithm and logic rules. We note that our automatic eye blink detection method is only a proof of concept and that further development need to be introduced before it can be used reliably in the context of blink types automatic classification. These further developments notably include a more varied data used in the training of the CNN models.