Semantic segmentation of hands in multimodal images: A region new-based CNN approach

Segmentation of body parts is a critical but challenging stage in the medical image processing pipeline due to the anatomical complexity. Recent advances in deep learning have successfully dealt with this complexity in visible images. However, the efficacy of these segmentation techniques for other modalities of interests such as X-ray images is not known. We propose a unified semantic segmentation approach of body parts for both X-ray and visible images which can be concurrently applied to both modalities. Unifying the two modalities in a single model not only reduces the number of parameters, but also improves the accuracy by enabling end-to-end training and inference. Quantitative results are validated on two clinical applications: (1) a static analysis of hand segmentation in visible and X-ray images; and (2) a dynamic analysis which quantifies and classifies epileptic seizures from clinical manifestations of visible hand and finger motions. The proposed model is a potential stepping stone towards developing more robust automated systems that support the assessment of medical conditions based on multimodal information.