Skip to Main content Skip to Navigation

Smart atlas for endomicroscopy diagnosis support: a clinical application of content-based image retrieval

Abstract : Probe-based Confocal Laser Endomicroscopy (pCLE) enables in vivo microscopic imaging of the epithelium during ongoing endoscopy, in situ and at real-time frame rate. Thanks to this novel imaging system, the endoscopists have the opportunity to perform non-invasive "optical biopsies". Traditional biopsies result in histological images that are usually diagnosed ex vivo by pathologists. The in vivo diagnosis of pCLE images is therefore a critical challenge for the endoscopists who typically have only little pathology expertise. The main goal of this thesis is to assist the endoscopists in the in vivo interpretation of pCLE image sequences. When establishing a diagnosis, physicians typically rely on similarity-based reasoning. To mimic this process, we explore content-based image retrieval (CBIR) approaches for diagnosis support. Our primary objective is to develop a system which automatically extracts several videos that are visually similar to the pCLE video of interest, but that are annotated with metadata such as textual diagnosis. Such a retrieval system should help the endoscopist in making an informed decision and therefore a more accurate pCLE diagnosis. For this purpose, we investigate the Bag-of-Visual-Words (BoW) method from computer vision. Analyzing the image properties of pCLE data leads us to adjust the standard BoW method. Not only single pCLE images, but full pCLE videos are retrieved by representing videos as sets of mosaics. In order to evaluate the methods proposed in this thesis, two different pCLE databases were constructed, one on the colonic polyps and one on the Barrett's esophagus. Due to the initial lack of a ground-truth for CBIR of pCLE, we first performed an indirect evaluation of the retrieval methods, using nearest-neighbor classification. Then, the generation of a sparse ground-truth, containing the similarities perceived between videos by multiple experts in pCLE, allowed us to directly evaluate the retrieval methods, by measuring the correlation between the retrieval distance and the perceived similarity. Both indirect and direct retrieval evaluations demonstrate that, on the two pCLE databases, our retrieval method outperforms several state-of-the-art methods in CBIR. In terms of binary classification, our retrieval method is shown to be comparable to the offline diagnosis of human expert endoscopists on the Colonic Polyps database. Because establishing a pCLE diagnosis is an everyday practice, our objective is not only to support one-shot diagnosis but also to accompany the endoscopists in their progress. Using retrieval results, we estimate the difficulty to interpret a pCLE video. We show that there is a correlation between the estimated difficulty and the diagnosis difficulty experienced by multiple endoscopists. The proposed difficulty estimator could thus be used in a self-training simulator, with difficulty level selection, which should help the endoscopists in shortening their learning curve. The standard visual-word-based distance already provides adequate results for pCLE retrieval. Nevertheless, little clinical knowledge is embedded in this distance. By incorporating prior information about the similarity perceived by pCLE experts, we are able to learn an adjusted visual similarity distance which we prove to be better than the standard distance. In order to learn pCLE semantics, we then leverage multiple semantic concepts used by the endoscopists to describe pCLE videos. As a result, visual-word-based semantic signatures are built which extract, from low-level visual features, a higher-level clinical knowledge that is expressed in the endoscopist own language.
Document type :
Complete list of metadatas
Contributor : Bibliothèque Mines Paristech <>
Submitted on : Monday, November 14, 2011 - 2:10:54 PM
Last modification on : Friday, October 23, 2020 - 4:59:26 PM
Long-term archiving on: : Monday, December 5, 2016 - 10:16:08 AM


  • HAL Id : pastel-00640899, version 1


Barbara André. Smart atlas for endomicroscopy diagnosis support: a clinical application of content-based image retrieval. Medical Imaging. École Nationale Supérieure des Mines de Paris, 2011. English. ⟨NNT : 2011ENMP0032⟩. ⟨pastel-00640899⟩



Record views


Files downloads