Deep Learning for Automatic Segmentation of Pituitary Adenomas: A Videomics Study

Background and objectives: Videomics, which integrates video-endoscopy and artificial intelligence, presents significant potential for real-time surgical analysis. Accurate intraoperative segmentation of pituitary adenomas (PAs) is crucial in endoscopic surgery to improve surgical precision. This study evaluates the performance of different deep learning (DL) models, namely the Swin Transformer, you only look once (YOLO), and Mask R-CNN, for automated PA segmentation, focusing on improving the accuracy of tumor boundary delineation. Methods: This retrospective study involved patients who underwent endoscopic endonasal surgery for confirmed PAs from January 2022 to December 2023. A Data set of 700 representative frames was extracted. Two clinicians manually segmented the frames (inter-rater reliability of κ = 0.85). The Data set was split into 70% for training (from 14 videos), 15% for validation (from 3 videos), and 15% for testing (from 3 videos). YOLO, Mask R-CNN, and Swin Transformer models were trained and optimized for 100 epochs using mean Average Precision (mAP) as primary metric. Kruskal-Wallis H-test was used for overall comparisons (P < .05), with pairwise Mann-Whitney U tests for detailed comparisons between models. Results: The Swin Transformer model achieved superior segmentation performance, with a test segmentation mAP[0.50] of 0.607, significantly outperforming YOLOv8x (mAP[0.50] = 0.416; P = .02, 95% CI: [0.56-0.65]) and Mask R-CNN (mAP[0.50] = 0.480; P = .04, 95% CI: [0.57-0.64]). The Swin Transformer model's Dice Similarity Coefficient was 0.89 (P < .01, 95% CI: [0.86-0.92]), compared with 0.83 for YOLOv8x and 0.81 for Mask R-CNN. The Swin Transformer also displayed optimal recall (0.91, P < .05, 95% CI: [0.88-0.94]) and precision (0.88, P < .05, 95% CI: [0.85-0.91]). Conclusion: The Swin Transformer model demonstrated the highest accuracy in PA boundary delineation among tested models, underscoring its potential as an advanced tool for intraoperative PA segmentation in endoscopic endonasal surgery.