Task
- Detect mitotic figures (cells undergoing cell division) from histopathology images (object detection).
- You will be provided with images scanned by 4 different scanners, 3 out of which are labeled. In total the set consists of 200 cases of breast cancer.
- Evaluation will be done on four scanners (two new, two are part of the training set) with the F1 score as main metric.
Important Dates
- August 15th: Preliminary test set available
- August 21th: Deadline for registration of participants
- Sept 2nd: Deadline for docker container submission and preprint abstract submission (extended)
- Oct 1st: Announcement of results at MICCAI 2021
How to participate
- Register on the challenge website and download the data set.
- Submit docker container(s) with your algorithm(s) (template provided here: https://github.com/DeepPathology/MIDOG_reference_docker).
- Provide a short paper (preprint) about your method.
- Note: Only your own work will qualify as a submission.
Challenge format
From the challenge report (Aubreville et al., Medical Image Analysis, 2023):
The challenge was held in conjunction with the 2021 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) conference as a one-time event. The structured challenge description (Aubreville et al., 2021) was accepted after a single-blind peer review process. Participants were able to register and obtain the training data five months prior to the challenge submission deadline, allowing for sufficient time to develop and evaluate their algorithms. After registration, the participants were provided with the training set data and a description of the images and annotations, together with a Jupyter notebook showcasing how to work with the data and how to train an example object detector (RetinaNet) with it.
The task of the challenge was the detection of mitotic figures on regions of interests (ROIs) of a predefined size. For the training set, 200 cases of breast cancer (representing 200 patients) were retrieved, the ROI was selected by a trained pathologist and the WSIs were digitized using four different WSI scanners (50 cases each). The test set consisted of 80 cases digitized by four scanners (20 each) out of which two scanners were part of the training set and two were unseen. The challenge dataset represents a good trade-off between capturing the naturally occurring variability of WSIs and time invested for annotation. The number of cases per scanner allows for a realistic estimation of the performance.
Due to the task being about generalization, prior knowledge about the images of the test set needed to be excluded. Thus, the participants had no knowledge of the nature of the test scanners, no access to the test images and had to submit a Docker container to be evaluated automatically on the test data. For this, a reference algorithm (Wilm et al., 2022) embedded into a Docker container was made available to the participants alongside a textual description and video tutorials on how to use it. All participating teams submitted their fully automatic algorithm containers on the grand-challenge.org website. To check container functionality and algorithmic validity, a preliminary test set (20 cases, 5 per scanner, same scanners as in the test set) was made available for automatic evaluation on the platform, two weeks prior to the submission deadline. The evaluation container for the challenge was also made available on github. Participants were not permitted to use other sources of images to train their models, besides general purpose datasets such as ImageNet.
In order to prevent overfitting to the characteristics of the preliminary test seft, the number of submissions was limited to one per day. After 15 days, the preliminary test phase was closed and the final test phase was started, to which the participants were able to only submit once. Alongside with the submission for evaluation on the final test set, participating teams had to provide a short paper, describing their approach and their preliminary results, on a publicly available pre-print server. Further details about the challenge, including details about the submission instructions, the publication policy and the timetable can be found in the structured challenge description (Aubreville et al., 2021).
Phases
Summarised from the text above (challenge report):
| Phase | Test data | Submissions |
|---|---|---|
| Preliminary test phase | 20 cases, 5 per scanner, same scanners as in the test set | limited to one per day; closed after 15 days |
| Final test phase | 80 cases digitized by four scanners (20 each) | only once |
Evaluation
From the challenge report, section 2.6 "Evaluation methods":
All participants were required to submit their approach as a docker container to the grand-challenge.org platform. There, the containers were automatically evaluated for each image of the test set independently. The participants had no access to any of the test images during the challenge and detailed detection results were also not available to them. This was done in order to ensure a fair comparison of the approaches and to discourage overfitting or manual fine-tuning towards the test set.
To ensure proper functionality of the automatically evaluated container images, we provided evaluation results on the preliminary test set. The main metrics ($F_{1}$ score, precision and recall) of submitted approaches were made available on a public leaderboard.
For the overall rating, the $F_{1}$ score was the primary metric. We calculated the $F_{1}$ score over all $N$ processed slides as
$$F_{1}=\frac{2\sum_{k}^{N}\mathrm{TP}{k}}{2\sum{k}^{N}\mathrm{TP}{k}+\sum{k}^{N}\mathrm{FN}{k}+\sum{k}^{N}\mathrm{FP}_{k}}$$
where $\mathrm{TP}{k}$, $\mathrm{FN}{k}$, and $\mathrm{FP}_{k}$ are the number of true positive, false negative and false positive detections on slide $k$ and $N$ is the total number of slides ($N=80$ for the final test set and $N=20$ for the preliminary test set).
The $F_{1}$ score was chosen as the main evaluation metric because it is defined as harmonic mean of precision and recall, and both underestimation as well as overestimation of the MC are equally severe for the diagnostic process. We did opt to calculate the overall $F_{1}$ instead of the mean $F_{1}$ of all slides, since slides with a low prevalence of mitotic figures would be overrepresented in this average.
We defined a detection to be a true positive whenever the Euclidean distance between a mitotic figure annotation and the detection was less than $7.5,\mu m$. This value corresponds to the average size of mitotic figures in our dataset and provides a reasonable tolerance for misalignment of detection and ground truth labels. All detections not within $7.5,\mu m$ of a ground truth mitotic figure annotation were considered false positives. Multiple detections of an already detected object were also counted as false positives, since they would introduce a positive bias to the MC. All ground truth mitotic figures without a detection within a proximity of $7.5,\mu m$ are considered false negatives.
Publications and Prizes
- All participants of the MIDOG challenge may submit a short LNCS Springer paper (up to 4 pages) with a deadline of three weeks after the workshop, which will be subject to peer review.
- Top performing teams will be invited to contribute to a challenge overview paper, which will be submitted to a high impact journal (MedIA/TMI).
- Best contribution (as of F1 score on test set) will receive 500€, second 300€ and third 200€.