Multi-view feature engineering for day-to-day joint clustering of multiple traffic datasets

A common task in traffic data analysis and management is categorizing different days based on similarities in their network-wide traffic states. Given the multifaceted nature of traffic, it is essential to consider multiple attributes for a comprehensive quantification. However, challenges arise when combining these attributes to achieve consistent day-to-day classification across datasets. While various data-driven classification algorithms have been proposed in traffic literature, challenges persist. These include a) applicability limited to univariate datasets, b) incompatibility with datasets containing missing values, c) distance concentration problem in high-dimensional clustering, d) inability to classify outliers, and e) computationally expensive hyperparameter optimization. This research introduces the MCMD (Multi-view Classification based on Consensus Matrix Decomposition) framework, a novel approach for the joint classification of multi-view traffic data. MCMD treats multiple traffic datasets with varying geographical coverage as complementary views of the entire network's traffic state. It then extracts shared hidden features across these datasets and assigns each day classification labels that are consistent across views. MCMD consists of three key modules: the novel Multi-view Uni-orthogonal Non-negative Matrix Factorization (MUNMF) algorithm, an outlier removal module, and the Ordering Points to Identify the Clustering Structure (OPTICS) algorithm. A logical integration of the above-stated modules enables MCMD to a) output scale-invariant (SI) and scale-variant (SV) classifications and b) identify outlier days based on the shape and scale of multi-view traffic-state profiles. Compared to existing clustering methods, the design of the MCMD algorithm offers greater versatility in handling both single- and multi-view datasets for SI and SV clustering, computational robustness to missing data, and resilience to the “distance concentration problem” associated with the curse of dimensionality. These advantages stem from its ability to extract relevant cross-dataset features, reduce dimensionality, and eliminate redundancy. Although the primary motivation of the framework is derived from the need to develop a traffic pattern repository to support reliable prior Origin-Destination (OD) selection for online dynamic OD demand adjustment, the paper, through extensive experiments on real-world and synthetic traffic datasets, demonstrates the effectiveness of MCMD from several generic standpoints. These include a) demonstration of the meaningfulness of SI and SV labels, b) assessment of the robustness toward missing information, c) evaluation of its effectiveness in classifying days with special events, d) benchmarking properties against alternative joint day-to-day clustering algorithms, and e) demonstrating the efficacy of the proposed hyperparameter selection method for efficient joint classification of multiple large-scale traffic datasets.