Developing a direct driver sleepiness detection system

0
209


Introduction

Driving while fatigued plays a major role in traffic accidents and results in enormous economic loss.1 Death and injuries from traffic accidents have increased despite the proliferation of modern driving technologies and road safety strategies.2 Thailand ranks first for traffic accident deaths among the ASEAN nations and third highest worldwide.3,4 The three main causes of traffic accidents are speed violations, illegal overtaking, and human errors such as drowsy driving.5,6

Mitigation for “drowsy driving” related death and injury can be achieved by improving early drowsiness detection for drivers and further developing driver alert systems. Previous studies have detected drowsiness by using brainwaves measured with an electroencephalogram (EEG),7,8 heart rate monitoring,9 and steering grip pressure monitoring.10 Moreover, brain–computer interfaces have been used to develop detection systems for drowsiness in laboratory simulations using brainwave monitoring with an EEG or eye movement with an electrooculogram (EOG).11 Additionally, a wide range of artificial intelligence (AI) techniques including artificial neural networks (ANNs) have been used to detect levels of drowsiness.10,12 Researchers have also developed alert systems by measuring face and eyelid aspect ratios,13 eye area,14 vehicle speed, steering angle12; examining facial expressions and eye blinking ratio13,15–25; lane departures, fatigue levels and video monitoring.26–29

The sensitivity and specificity of many approaches have been studied30,31 in comparison with the lane-crossing and microsleep models with a variety of results.25,32–35 Most of these methods have been tested for improving detection systems; however, variables such as hypoglycemia caused by fasting, body movements, light interference, consumption of sedatives, and having oily hair have interfered with EEG readings. Movements such as swallowing and blinking have especially caused inaccuracies.36 Finally, although smartphones are common, installation of drowsiness detection applications require the devices to be set up for a particular individual’s eyes and facial features.37 The feasibility of these existing detection approaches is limited by their cost and practicality.

This study aims to develop an algorithm for the detection of drowsiness using alternated-facial-expression data with microsleep-starting-point data provided by EEG, then measure the efficacy of our algorithm and improve upon it by feeding results back into our model. The change in alertness observed by facial expression is based on the eye aspect ratio and the facial-drowsiness-expression-algorithm. The advantage of this development is that the system can be used for warning drivers of potential dangers posed by their fatigue and encourage them to stop driving, fostering accident prevention.

Materials and Methods

The Study Workflow

First, EEG signals and video capture data of the subject’s face were simultaneously collected. The EEG signals were then interpreted by a somnologist to identify the subject’s state of drowsiness, especially to mark the time points of the microsleep initiation. Meanwhile, the subject’s facial data and time stamps were extracted from the video for use as a classifier of the microsleep developments to identify the state of drowsiness automatically within our model. Finally, the performance of the classifier was evaluated. The details of each component in the workflow are described as follows (Figure 1).

Figure 1 Workflow of the study.

Participants

This study included 10 healthy subjects (3 men and 7 women) who met the inclusion criteria including 1) age 18–50 years, 2) no history of seizure, 3) no hypnotic drug intake within 4 weeks of the study enrollment, and 4) no abnormal eye movement. Written informed consent was obtained from each subject before conducting the study, and the experimental procedures were carried out in compliance with the local Research Ethics Committee for research on human subjects.

EEG Signal Collection and Interpretation

All subjects were screened through a clinical interview and enrollment process. They were instructed to relax on a chair while resting their chin on an individually fitted headrest, and to stare at an application for 2 hours or until the EEG showed an absence of alpha waves for 5 minutes consecutively. The room environment was intended for comfort and to isolate any external factors that may affect the EEG signals. Constant temperature (68–76°F), and light intensity, approximately 500–1000 lux, were maintained.38,39 Thirteen EEG channels were configured following the International 10/20 System, including the following: two frontal brain regions (F3, F4), two central brain regions (C3, C4), two occipital brain regions (O1, O2), two reference channels (A1, A2) placed on the mastoid process, two EOG channels recording the corneo-retinal potential difference, and three electromyography (EMG) channels (Figure 2).40–42 Each electrode had a built-in impedance of 5 kΩ, and the electrode-skin impedance was maintained below 5 kΩ by utilizing abrasive electrode paste.41,43,44 Online EEG was acquired in real-time monitoring of 30 seconds in each epoch that enabled identification of the sleep stage, typically with the commercial software (Polysmith PSG-1100 Nihon kohden software). The software was configured with a minimum digital resolution of 12 bits per sample, sampling rates of 200–500 Hz, filter setting of 0.3 Hz by the low-frequency filter and 35 Hz by the high-frequency filter in EEG and EOG, 10 Hz by the low-frequency filter and 100 Hz by the high-frequency filter in EMG were recorded.45,46

Figure 2 Electroencephalography electrodes placement locations on the scalp according to the international 10–20 system, whereas the electrical signals from the white electrodes are used for defining sleep–wake stage.

The sleep stages use the standard EEG waveforms criteria for interpretation which include delta (0.5 to 4Hz), theta (4 to 7Hz), alpha (8 to 12Hz), and beta (13 to 30Hz).41,47,48 Conventionally, sleep onset is demonstrated by the transition of EEG alpha activity (8–12 Hz) to theta activity (4–7 Hz), but the first stage of sleep needs the presence of theta and vertex waves to be more than 50% of each individual EEG epoch.49–51 The termination of sleep or drowsiness indicated at EEG repetitiveness of alpha activity and absence of theta activity and delta activity. Theta detection during wakefulness was related to drowsiness or microsleep for active subjects with their eyes open.52–54 In this study, a certified sleep technician recorded the specific time that the EEG showed the onset of microsleep, which was specified by the transition of EEG alpha activity to the presence of theta activity and used it to determine the onset of drowsiness (Figure 3).

Figure 3 Polysomnography epochs display for 30 seconds time-range. Red arrow illustrates the specific microsleep time point.

Video Capture Device

The device was built with a Raspberry Pi 4, which operates as a microcontroller for video data processing and alert triggering. The camera module is connected to a camera serial interface on the Raspberry Pi using a ribbon cable. In our experiment, a video capture device was installed to frontally record the participant’s face. Both EEG signal collection and the video capture device were initiated simultaneously.

Facial Information Extraction

The facial information of each subject was extracted from facial components used for feature vector construction. For each frame of the given volunteer’s face capture data, facial landmarks (illustrated in Figure 4) were detected by using an ensemble of regression trees.55 Four facial features of the frame namely eye aspect ratio (EAR), mouth aspect ratio (MAR), face length (FL), and face width balance (FWB) were then calculated as follows:

(1)


(2)


(3)


(4)

Figure 4 Visualizing the 68 facial landmark coordinates on an AI-generated face image.

where is the Euclidean distance between landmark points and .

The EAR represents the behavior of blinking as it depicts the ratio of height to width of eyes. The of zero indicates the person is blinking. The behavior of speech or yawning is captured by the MAR that presents the ratio of height to width of mouth. The of zero implies that the person is speaking or yawning. The facial movement is encapsulated by FL and FWB. The FL depicts the length between the chin and plane of eyes. The decrease and increase of indicates that the person is nodding. Meanwhile, the FWB pictures the difference of widths of left and right sides of the face. The decrease and increase of imply that the person is turning his face. Each frame on the input video has extracted the EAR, MAR, FL, and FWB. Consequently, for a given video, we concluded with the time series of EARs, MARs, FLs, and FWBs. For a frame in the video, a feature vector was constructed by combining the facial features between and . Several combinations of facial features were investigated. We adopted the DFT to convert each facial feature to the frequency domain.

Drowsiness State Identification

To determine the drowsiness state of the person at the frame , an ANN was adopted as the classifier. As shown in Figure 5, the ANN is composed of three layers including input, hidden, and output layers. The facial features of 50 frames (2 seconds) were pressed to the ANN through the input layer. The hidden layer consists of neural, where and were the size of input and output layers, respectively. A sigmoid function was adopted as the activation function of the hidden layer. The output layer of the ANN has one neuron that produces a probability of drowsiness state at the frame t. The value of the probability varies between 0 and 1. The value of 0 indicates the person is active, while the value of 1 indicates the person is in sleep state at the frame .

Figure 5 Architecture of ANN for identifying drowsiness state.

Statistical Analysis

To evaluate detection performance of the proposed method, 10-fold cross-validation (10-CV) was performed. The 10-CV process was repeated ten times with each of the ten subjects used exactly once as the test dataset. In each fold, the data of nine subjects were used as a training dataset while the remaining subject was used as a test dataset. The proposed method correctly identified the status of the subject at a time point t if at least one EEG signal within t±3 seconds indicated the same status. The delay and early warning of drowsiness state were accepted within 6 seconds. Consequently, true positive, true negative, false positive, and false negative were defined in Table 1.

Table 1 The Definition of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) for Evaluate the Performance of Real-Time Detecting Drowsiness State at a Point of Time

The detection performance was measured in terms of accuracy, sensitivity, and specificity. Good performance of any classification is defined by high accuracy, high sensitivity, and high specificity. To select the proper model, the averaged values of the accuracy, sensitivity, and specificity of 10-CV were considered.

Results

Having compared the results of the ANN models with four composite features, (EAR and MAR; EAR, MAR, and FL; EAR, MAR and FWB; EAR, MAR, FL, and FWB), the ANN model with EAR and MAR features had the most sensitivity (70.12%) while the ANN model with EAR, MAR, and FL features had the most accuracy and specificity (60.76% and 58.71%, respectively). The ANN model with EAR and MAR features had the highest sensitivity (72.25%) while the ANN model with EAR, MAR, and FL features had the highest accuracy and specificity (60.40% and 54.10%, respectively) using the ANN and DFT model with four composite features (Table 2).

Table 2 The Performance of the ANN with Various Features Compared with ANN and DFT with Various Features (in Seconds)

The ANN model with EAR and MAR had the highest sensitivity (64.97%) while the ANN model with EAR, MAR, and FL features had the highest accuracy and specificity (63.15% and 62.67%, respectively) for results of the ANN models with four composite features. The features with the highest sensitivity, accuracy, and specificity were the ANN model for results of the ANN and DFT models with four composite features (Table 3).

Table 3 The Performance of the ANN with Various Features Compared with ANN and DFT with Various Features (in Frame)

Discussion

Fatigue in drivers can be observed by both direct driver inspection and indirectly, however, sleepy drivers indicate fatigue in different ways and can be affected by various driving conditions and environments. Importantly, driving behaviors detected indirectly are pre-specified differently by each automobile manufacturer which are lacking in clinical validation.

The inspection of biophysiological information may focus specifically on alertness levels such as our study and will provide the opportunity to develop a portable device, which can be used in more than one vehicle. During the development process, our system learned the dynamic changes of pre-specified facial points and areas in the form of qualitative and quantitative data for interpretation. The guidance from the EEG gave the specific time-point of microsleep initiation to the system learning. Finally, the accuracy of the system was validated back with the EEG again.

Limitations

Limitations of our algorithm include the need for the preset position (the distance between the camera and the face of the subject) and the inability of users to wear sunglasses while driving. Further validation for real-world variables is needed.

Conclusion

The ANN with DFT combined with the EAR, MAR, and FL offered the best performance. Our direct driver sleepiness detection system developed from the integration of biophysiological information and internal validation provide a valuable algorithm, specifically toward alertness level.

Data Sharing Statement

The data that support the findings of this study are available from the corresponding author on reasonable request.

Ethical Approval and Participation Consent

This study received approval from the Research Ethics Committees, Faculty of Medicine, Chiang Mai University (REC no. 386/2019) and followed to GCPs and relevant international ethical guidelines, applicable laws, and regulations including Declaration of Helsinki. All participants gave consent after being informed about the aim of the study as well as their right to refuse to participate.

Acknowledgments

We are incredibly grateful for all participants in this study. In addition, we would like to thank Chiang Mai University for partial support.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Funding

This research was supported by The Faculty of Medicine, Chiang Mai University, grant number 039/2563.

Disclosure

The authors report no conflicts of interest in this work.

References

1. World Health Organization. Review of Thailand’s status against voluntary global targets for road safety risk factors and service delivery mechanism; 2020.

2. Central Information Technology Center. Road crash case statistics. Available from: www.pitc.police.go.th//2014. Accessed October 19, 2014.

3. World Health Organization. Global status report on road safety 2015; 2015. Available from: http://www.who.int/violence_injury_prevention/road_safety_status/20.15/en/ Accessed September 7, 2022.

4. Supalaknari S, Hintao K. Factors affecting the competency of road crash investigation in the context of Thai police. Humanit Arts Soc Sci Stud. 2018;18(2):429–442.

5. Klinjun N, Kelly M, Praditsathaporn C, Petsirasan R. Identification of factors affecting road traffic injuries incidence and severity in Southern Thailand based on accident investigation reports. Sustainability. 2021;13(22):12467. doi:10.3390/su132212467

6. Sinlapabutra T. Current Situation of Road Safety in Thailand. Available from: https://www.unescap.org/sites/default/files/2.23.Thailand-1.pdf. Accessed September 7, 2022.

7. Lopez de la OJ, Ibáñez NR, González MN, et al. Development of a system to test somnolence detectors with drowsy drivers. Procedia Soc Behav Sci. 2012;48:2058–2070. doi:10.1016/j.sbspro.2012.06.1179

8. Faber J. Detection of different levels of vigilance by EEG pseudo spectra. Neural Netw World. 2004;14:285–290.

9. Sun Y, Yu XB. An innovative nonintrusive driver assistance system for vital signal monitoring. IEEE J Biomed Health Inform. 2014;18(6):1932–1939. doi:10.1109/jbhi.2014.2305403

10. Chieh TC, Mustafa M, Hussain A, Zahedi E, Majlis B. Driver fatigue detection using steering grip force. Proceedings Student Conference on Research and Development, 2003 SCORED 2003; 2003: 45–48.

11. Arnin J, Anopas D, Horapong M, et al. Wireless-based portable EEG-EOG monitoring for real time drowsiness detection. IEEE; 2013:4977–4980.

12. Vasudevan S, Anudeep J, Kowshik G, Nair P. An AI approach for real-time driver drowsiness detection—A novel attempt with high accuracy; 2021:305–316.

13. Sahayadhas A, Sundaraj K, Murugappan M. Detecting driver drowsiness based on sensors: a review. Sensors. 2012;12(12):16937–16953. doi:10.3390/s121216937

14. Devi MS, Bajaj PR. Driver fatigue detection based on eye tracking. Presented at: Proceedings of the 2008 First International Conference on Emerging Trends in Engineering and Technology; 2008. doi:10.1109/ICETET.2008.17.

15. Kircher A, Uddman M, Sandin J. Vehicle control and drowsiness. In: VTI Meddelande 922A; 2002.

16. Vural E, Cetin M, Ercil A, Littlewort G, Bartlett M, Movellan J. Drowsy driver detection through facial movement analysis. International Workshop on Human-Computer Interaction; 2007:6–18.

17. Danisman T, Bilasco I, Djeraba C, Ihaddadene N. Drowsy driver detection system using eye blink patterns. Presented at: International Conference on Machine and Web Intelligence; 2010.

18. Biswal AK, Singh D, Pattanayak BK, Samanta D, Yang M-H. IoT-based smart alert system for drowsy driver detection. Wirel Commun Mob Comput. 2021;2021:6627217. doi:10.1155/2021/6627217

19. van der Wall H, Doll R, van Westen G, et al. Using machine learning techniques to characterize sleep-deprived driving behavior. Traffic Inj Prev. 2021;22(5):366–371. doi:10.1080/15389588.2021.1914837

20. Mehta S, Dadhich S, Gumber S, Jadhav Bhatt A. Real-time driver drowsiness detection system using eye aspect ratio and eye closure ratio; 2019.

21. Dua HK, Goel S, Sharma V. Drowsiness detection and alert system; 2018:621–624.

22. Jacobé de Naurois C, Bourdin C, Bougard C, Vercher J-L. Adapting artificial neural networks to a specific driver enhances detection and prediction of drowsiness. Accid Anal Prev. 2018;121:118–128. doi:10.1016/j.aap.2018.08.017

23. Wang X, Xu C. Driver drowsiness detection based on non-intrusive metrics considering individual specifics. Accid Anal Prev. 2016;95:350–357. doi:10.1016/j.aap.2015.09.002

24. Jacobé de Naurois C, Bourdin C, Stratulat A, Diaz E, Vercher J-L. Detection and prediction of driver drowsiness using artificial neural network models. Accid Anal Prev. 2019;126:95–104. doi:10.1016/j.aap.2017.11.038

25. Liang Y, Horrey WJ, Howard ME, et al. Prediction of drowsiness events in night shift workers during morning driving. Accid Anal Prev. 2019;126:105–114. doi:10.1016/j.aap.2017.11.004

26. Forsman PM, Vila BJ, Short RA, Mott CG, Van Dongen HP. Efficient driver drowsiness detection at moderate levels of drowsiness. Accid Anal Prev. 2013;50:341–350. doi:10.1016/j.aap.2012.05.005

27. Grace R, Steward S. Drowsy driver monitor and warning system. Iowa Research Online; 2001:64–69.

28. Grace R, Byrne VE, Bierman D, et al. A drowsy driver detection system for heavy vehicles. 17th DASC AIAA/IEEE/SAE Digital Avionics Systems Conference Proceedings (Cat No98CH36267); 2, 1998:I36/1–I36/8.

29. Moller HJ, Kayumov L, Bulmash EL, Nhan J, Shapiro CM. Simulator performance, microsleep episodes, and subjective sleepiness: normative data using convergent methodologies to assess driver drowsiness. J Psychosom Res. 2006;61(3):335–342. doi:10.1016/j.jpsychores.2006.04.007

30. Sparrow AR, LaJambe CM, Van Dongen HPA. Drowsiness measures for commercial motor vehicle operations. Accid Anal Prev. 2019;126:146–159. doi:10.1016/j.aap.2018.04.020

31. Watling CN, Mahmudul Hasan M, Larue GS. Sensitivity and specificity of the driver sleepiness detection methods using physiological signals: a systematic review. Accid Anal Prev. 2021;150:105900. doi:10.1016/j.aap.2020.105900

32. Barua S, Ahmed MU, Ahlström C, Begum S. Automatic driver sleepiness detection using EEG, EOG and contextual information. Expert Syst Appl. 2019;115:121–135. doi:10.1016/j.eswa.2018.07.054

33. Guo M, Li S, Wang L, Chai M, Chen F, Wei Y. Research on the relationship between reaction ability and mental state for online assessment of driving fatigue. Int J Environ Res Public Health. 2016;13(12):1174. doi:10.3390/ijerph13121174

34. Mårtensson H, Keelan O, Ahlström C. Driver sleepiness classification based on physiological data and driving performance from real road driving. IEEE Trans Intell Transp Syst. 2019;20(2):421–430. doi:10.1109/TITS.2018.2814207

35. Bundele MM, Banerjee R. ROC analysis of a fatigue classifier for vehicular drivers; 2010:296–301.

36. Hopkins J. Electroencephalogram (EEG). The Johns Hopkins University. Available from: https://www.hopkinsmedicine.org/health/treatment-tests-and-therapies/electroencephalogram-eeg. Accessed July 11, 2021.

37. Mohammad F, Mahadas K, Hung GK. Drowsy driver mobile application: development of a novel scleral-area detection method. Comput Biol Med. 2017;89:76–83. doi:10.1016/j.compbiomed.2017.07.027

38. Light and lighting, Lighting of work places – Indoor work places. BS EN. 2011;12464-1. https://knowledge.bsigroup.com/products/light-and-lighting-lighting-of-work-places-indoor-work-places/standard.

39. Kruisselbrink T, Dangol R, Rosemann A. Photometric measurements of lighting quality: an overview. Build Environ. 2018;138:42–52. doi:10.1016/j.buildenv.2018.04.028

40. Jasper H. Report of the committee on methods of clinical examination in electroencephalography. Electroencephalogr Clin Neurophysiol. 1958;10:370–375.

41. IG Campbell. EEG recording and analysis for sleep research. Curr Protoc Neurosci. 2009. doi:10.1002/0471142301

42. Munday JA. Instrumentation and electrode placement. Respir Care Clin N Am. 2005;11(4):605–615. doi:10.1016/j.rcc.2005.08.004

43. Kappenman ES, Luck SJ. The effects of electrode impedance on data quality and statistical significance in ERP recordings. Psychophysiology. 2010;47(5):888–904. doi:10.1111/j.1469-8986.2010.01009.x

44. Górecka J, Makiewicz P. The dependence of electrode impedance on the number of performed EEG examinations. Sensors. 2019;19(11):2608. doi:10.3390/s19112608

45. Pipberger HV. Recommendations for standardization of leads, and of specifications for instruments in electrocardiography and vectorcardiography.Report of the Committee on Electrocardiography. Am Heart Assoc. 1975;52:11–31.

46. Kreuzer M, Polta S, Gapp J, Schuler C, Kochs EF, Fenzl T. Sleep scoring made easy—Semi-automated sleep analysis software and manual rescoring tools for basic sleep research in mice. MethodsX. 2015;2:232–240. doi:10.1016/j.mex.2015.04.005

47. CS Nayak, AC Anilkumar. EEG Normal Waveforms. StatPearls; 2021.

48. Carskadon MA, Dement WC. Monitoring and staging human sleep; Normal human sleep. In: Principles and Practice of Sleep Medicine. 5th ed; 2011:16–26.

49. Santamaria J, Chiappa KH. The EEG of drowsiness in normal adults. Neurophysiol. 1987;4:327–382.

50. Ogilvie R. The process of falling asleep. Sleep Med Rev. 2001;5(3):247–270. doi:10.1053/smrv.2001.0145

51. Hyoki K, Shigeta M, Tsuno N, Kawamuro Y, Kinoshita T. Quantitative electro-oculography and electroencephalography as indices of alertness. Electroencephalogr Clin Neurophysiol. 1998;106(3):213–219. doi:10.1016/S0013-4694(97)00128-4

52. Daniel RS. Alpha and theta EEG in vigilance. Percept Mot Skills. 1967;25(3):697–703. doi:10.2466/pms.1967.25.3.697

53. Horváth M, Frantik E, Kopriva K. EEG theta activity increase coinciding with performance decrements in a monotonous task. Act Nerve Super. 1976;18:207–210.

54. Torsvall L, Åkerstedt T. Sleepiness on the job: continuously measured EEG changes in train drivers. Electroencephalogr Clin Neurophysiol. 1987;66(6):502–511. doi:10.1016/0013-4694(87)90096-4

55. Kazemi V, Sullivan J. One millisecond face alignment with an ensemble of regression trees. 2014 IEEE Conference on Computer Vision and Pattern Recognition; 2014:1867–1874.



Source link