Oral tumor detection and localization using detection transformer (DETR) with multi-scale feature extraction and self-supervised learning

Abstract

This study presents a transformer-based framework for lesion-level detection and localization of oral tumors using the Detection Transformer (DETR) architecture under limited-data medical imaging conditions. The dataset consists of 131 clinically acquired oral images (87 cancerous, 44 non-cancerous). To prevent information leakage and inflated performance estimation, dataset partitioning was performed at the original-image level prior to augmentation, and a 5-fold stratified cross-validation protocol was implemented. Data augmentation was applied exclusively to training subsets, while evaluation was conducted only on non-augmented validation images. To enhance representation learning under low annotation availability, a SimCLR-based self-supervised pretraining stage was incorporated within each cross-validation fold using only training-fold images. The pretrained ResNet-50 backbone was then fine-tuned within the DETR framework for supervised tumor localization. Across cross-validation folds, the proposed model achieved a mean mAP of 0.496 ± 0.027, AP50 of 0.738 ± 0.031, and mean IoU of 0.638 ± 0.019. Bootstrap-based 95% confidence intervals were reported to quantify statistical uncertainty. External validation on an independent oral lesion dataset demonstrated consistent localization performance under domain shift, supporting generalization beyond the primary dataset. Ablation analysis confirmed that self-supervised initialization improves detection stability compared to random initialization. The results demonstrate that a leakage-free transformer-based detection framework combined with self-supervised representation learning can provide reliable lesion-level localization performance in low-resource oral cancer imaging scenarios.

keywords Tumor detection ,Detection transformer (DETR), Oral cancer localization, Self-supervised learning (SSL)Attention mechanism, Grad-CAM visualization, Image augmentation