Multimodal Physiological Ensemble Deep Learning for Driver's Takeover Reaction Time Classification in Automated Vehicles

Enabling automated vehicles to predict driver takeover readiness is critical for safe control transitions, yet most prediction research focuses on urgent rather than planned takeover scenarios. This study presents a framework to classify driver reaction times in responding to planned takeover requests (TOR) in SAE Level 3/4 automated vehicles. In a driving simulator study, participants (N = 42) completed three 14-minute drives, each concluding with a TOR upon exiting the Operational Design Domain, across three non-driving-related task conditions, yielding 126 TOR events. We compare deep learning (DL) architectures and physiological measures (i.e., gaze tracking, electrodermal activity, electrocardiogram) to classify the driver’s reaction times (time from TOR onset to first physical response) for fast and slow reaction classes. We propose an end-to-end multimodal ensemble DL framework for predicting driver’s takeover reaction times, leading to an accuracy of 88%. Results show that the fusion performance based on single-modality (i.e., gaze tracking) matches that of three-modality fusion (Matthews Correlation Coefficient, MCC = 0.63), while adding the electrocardiogram yields only marginal improvement (MCC = 0.67). This suggests camera-based driver monitoring systems—already standard in new vehicles—provide sufficient capability for planned takeover reaction time classification without combining with additional physiological sensors, though generalisation to urgent TOR scenarios warrants further study.