Precise partitioning of cervical spine cord and multiple sclerosis (MS) lesions is essential for medical assessment and therapy planning. This paper presents a novel approach combining reinforcement learning (RL) with multi-scale attention mechanisms in a U-Net architecture for automated medical image segmentation. Our method integrates a Deep Q-Network (DQN) agent that dynamically optimizes segmentation thresholds based on image characteristics, while multi-scale attention modules enhance feature representation at different spatial scales. The proposed RL-Attention U-Net underwent evaluation on the cervical spinal cord and MS lesion dataset, attaining avant-garde performance with a Dice coefficient of 0.9460, IoU of 0.8986, precision of 0.9507, and recall of 0.9430. Compared to existing methods, our approach demonstrates superior segmentation accuracy while maintaining computational efficiency with 14.90 images per second processing speed. The integration of reinforcement learning for adaptive threshold optimization and multi-scale attention for enhanced feature learning constitutes a substantial progression in medical image segmentation, particularly for challenging anatomical structures like cervical spinal cord and MS lesions.