Alzheimer’s disease (AD) is a common elderly illness, and the hand movement abilities of patients differ from those of normal individuals. Focusing on the utilization of RGB, optical flow, and hand skeleton as tri-modal image information for early AD recognition, a method for early AD recognition via multi-modal hand motion quality assessment (EADR) is proposed. First, a hybrid modality feature encoder incorporating global contextual information was designed to integrate the global contextual information of features from three specific modality branches. Subsequently, a fusion modality feature decoder network incorporating specific modality features was proposed to decode the overlooked information in the fusion modality branch from specific modality features, thereby enhancing feature fusion. Experiments demonstrated that EADR effectively could capture high-quality hand motion features and excelled in hand motion quality assessment tasks, outperforming existing models. Based on this, the action quality scoring regression model trained using the k-nearest neighbors algorithm demonstrated the best recognition performance for AD patients, with Spearman’s rank correlation coefficient and Kendall’s rank correlation coefficient reaching 90.98% and 83.44%, respectively. This indicates that the assessment of hand motor ability may serve as a potential auxiliary tool for early AD identification.