netidee
mAP SSD vs SSD-EDL
Current State and Next Steps
Discussing the current limitations and suggesting further directions for experiments (29.03.2025)
Förderjahr 2023 / Stipendien Call #18 / ProjektID: 6885 / Projekt: Increasing Trustworthiness of Edge AI by Adding Uncertainty Estimation to Object Detection

Current experiments showed severe degradations in performance due to a combination of limitations in object detection architecture, training routine, and uncertainty estimation methods.

In our interim report, we conducted the first experiments on adding uncertainty estimation through Evidential Deep Learning for classification [6] to the Single Shot Detector (SSD) [3], as seen in the figure below:

mAP SSD vs SSD-EDL

Results showed that the detector was able to filter out uncertain predictions, resulting in increased precision (fewer false positives), however, the recall and also the overall mean Average Precision (mAP) decreased, showing limitations of directly applying EDL to object detection.

In our following experiments, we took another shot at adapting EDL to SSD by implementing a newer approach that uses Beta distributions for modeling evidence [4], compared to the default Dirichlet distribution. While Beta distributions resulted in a better mAP score than Dirichlet, both EDL versions still lacked mAP compared to the base SSD detector without uncertainty estimation.

Limitations

The insights gained so far suggested limitations in several possible ways:

  • EDL is not suitable for the SSD architecture, where different detectors might be more compatible.
  • Training an EDL network from scratch is not feasible, instead, it needs to learn on top of a pre-trained object detector.
  • EDL needs to be further adapted to make it more compatible with object detection.
  • Although promising, EDL might not be the overall best method for (fast) uncertainty estimation in object detection, alternatives may surpass it.

Next Steps

We tackle all those points by restructuring our experimental setup from the ground up. First, the state-of-the-art YOLO11 [2] object detection framework will be chosen as a base, coming with up-to-date architecture and training routines. Second, transfer learning will be applied to learn uncertainty estimation on top of an already well-performing object detector. Third, for EDL implementation, we switch to the approach of Park et al. [5], which learns EDL on a separate branch in the model output (head), leaving the base object detection intact, allowing it to build up on the existing performance level and improve from there. Lastly, we also include a well-known uncertainty estimation approach in our evaluation, Monte-Carlo Dropout [1], with its modified version Monte-Carlo Dropblock being applied in object detection [7]. While the Dropout technique performs well at estimating the uncertainty of the predictions, its downside is the significant computational overhead, which we will have to balance in order to make it a feasible solution for real-time edge devices with low computational power. The next blog post will feature updates on the performance of those new implementations.

References

[1] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning, pages 1050–1059. PMLR, June 2016. ISSN: 1938-7228.

[2] Glenn Jocher and Jing Qiu. Ultralytics YOLO11, 2024. tex.orcid: 0000-0001-5950-6979, 0000-0002-7603-6750, 0000-0003-3783-7069.

[3] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single Shot MultiBox Detector. In Computer Vision – ECCV 2016, pages 21–37. Springer International Publishing, 2016. ISSN: 1611-3349.

[4] Tejas Pandey, Nick Pears, William A P Smith, and John Alexander McDermid. E-DETR: Evidential deep learning for end-to-end uncertainty estimation in object detection, 2025.

[5] Younghyun Park, Wonjeong Choi, Soyeong Kim, Dong-Jun Han, and Jaekyun Moon. Active learning for object detection with evidential deep learning and hierarchical uncertainty aggregation. In The eleventh international conference on learning representations, 2023.

[6] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in neural information processing systems, volume 31. Curran Associates, Inc., 2018.

[7] Sai Harsha Yelleni, Deepshikha Kumari, Srijith P.K., and Krishna Mohan C. Monte Carlo DropBlock for modeling uncertainty in object detection. Pattern Recognition, 146:110003, February 2024.

Tags:

Machine Learning Object Detection Edge computing Uncertainty Estimation Evaluation Artificial Intelligence

Daniel Hofstätter

Profile picture for user Daniel Hofstätter
CAPTCHA
Diese Frage dient der Überprüfung, ob Sie ein menschlicher Besucher sind und um automatisierten SPAM zu verhindern.

    Weitere Blogbeiträge

    Base object detection vs uncertainty estimation approaches

    YOLO with Uncertainty Estimation

    The current expirimental setup will be reworked from its foundation by switching from the SSD model to the latest YOLO11 object detector.
    mAP for SSD EDL Beta

    Evidential Deep Learning Tested on the Single-Shot Detector

    Evidential Deep Learning (EDL) for classification was added to the Single-Shot Detector (SSD). However, the evaluation shows a significant loss in precision. Therefore, further implementation approaches will be discussed.
    Active Learning Cycle Using Uncertainty Estimation

    Continual and Active Learning: A Use Case for Uncertainty Estimation in Object Detection

    Uncertainty estimation helps models to identify difficult inputs, which can be further used for continual learning, enabling models to evolve. Active learning prioritizes valuable data by utilizing uncertainty as a metric, reducing labeling costs.
    Precision-Recall curve showing mean average precision for object detection

    Evaluating the Quality of Uncertainty Estimation

    When adding uncertainty estimation to object detection, we need to make sure that these new modifications do not degrade the performance of the base model. Therefore, we evaluate the Precision-Recall curve and its mean Average Precision.
    Uncertainty Estimation Example Mockup

    Adding Uncertainty Estimation to Object Detection

    Object detection models typically do not capture the whole uncertainty in their predictions. It is important to first know what types of uncertainty there are, how they can be estimated, and what the impact on computational resources is.
    Object Detection Example Oxford Town Centre

    Uncertainty Estimation in Edge AI

    Current object detection models in machine learning, e.g., for detecting people in images, lack uncertainty expression, making predictions less trustworthy. This work evaluates uncertainty estimation approaches for efficient Edge AI.