Due to its irregular shape and varying contour, pancreas segmentation is a recognized challenge in medical image segmentation. Convolutional neural network (CNN) and Transformer-based networks perform well but have limitations: CNN have constrained receptive fields, and Transformer underutilize image features. This work proposes an improved pancreas segmentation method by combining CNN and Transformer. Point-wise separable convolution was introduced in a stage-wise encoder to extract more features with fewer parameters. A densely connected ensemble decoder enabled multi-scale feature fusion, addressing the structural constraints of skip connections. Consistency terms and contrastive loss were integrated into deep supervision to ensure model accuracy. Extensive experiments on the Changhai and National Institute of Health (NIH) pancreas datasets achieved the highest Dice similarity coefficient (DSC) values of 76.32% and 86.78%, with superiority in other metrics. Ablation studies validated each component’s contributions to performance and parameter reduction. Results demonstrate that the proposed loss function smooths training and optimizes performance. Overall, the method outperforms other advanced methods, enhances pancreas segmentation performance, supports physician diagnosis, and provides a reliable reference for future research.