The following is taken from the challenge report (Aubreville et al., "Mitosis domain generalization in histopathology images — The MIDOG challenge", Medical Image Analysis, 2023, arXiv:2204.03742).
Participation
Initially, 237 individuals registered on the challenge website and 161 users joined the challenge on the grand-challenge.org platform. Members of the organizers’ institutes were not allowed to parcitipate in the challenge. 46 users submitted at least one docker container to the challenge. At the end, 17 teams made a submission to the final test set. Single-blind peer review was carried out on all submitted short papers. After the peer review, we invited all teams that passed peer review and where the approach exceeded a minimum score of F1≥0.6 to the workshop (12/17 teams, acceptance rate 70.6%). The approaches presented in the workshop also are compared in this paper.
Final results
The majority of approaches were able to provide better results than the CNN baseline. The domain-adversarial reference method (Wilm et al., 2022) yielded a competitive F1 score of 0.718 on the test set and was outperformed by only four approaches (see Table 2). With an F1 score of 0.748, the overall best performance was reached by Yang et al., utilizing segmentation and Fourier-domain mixing as augmentation (Yang et al., 2022). It is worth noting that this approach was amongst the only approaches which was able to achieve good performance on Scanner D (together with the approach by Razavi et al., 2022), while other approaches performed better on other scanners. Even though the approach did not have the best performance on each individual scanner, it was the most consistently well-performing approach across all scanners, and thus the most generalizing solution. The runner-up approach (Jahanifar et al., 2022) had an almost identical F1 score compared to the best approach, supported also by a good overall performance across all scanners, with some minor weaknesses on scanner D, but it was the best performing algorithm on Scanner E. On Scanner F, the best performing approach was the multi-stage cascaded RCNN approach by Razavi et al. (2022), and on the scanner A, which was part of the training set, the best solution came from Fick et al. (2022).
Table 2: F1 score for all participating approaches
F1 score for all participating approaches. Numbers in square brackets indicate 95% confidence interval as determined by bootstrapping.
| Team | Overall | Scanner A | Scanner D | Scanner E | Scanner F |
|---|---|---|---|---|---|
| Reference | 0.718 [0.665,0.762] | 0.791 [0.673,0.818] | 0.708 [0.620,0.766] | 0.718 [0.631,0.811] | 0.593 [0.551,0.719] |
| CNN baseline | 0.698 [0.639,0.745] | 0.687 [0.650,0.812] | 0.700 [0.621,0.770] | 0.657 [0.519,0.747] | 0.606 [0.521,0.717] |
| Top5 ensemble | 0.773 [0.722,0.813] | 0.796 [0.748,0.874] | 0.745 [0.667,0.780] | 0.787 [0.744,0.861] | 0.642 [0.581,0.761] |
| AI medical | 0.748 [0.704,0.781] | 0.793 [0.729,0.830] | 0.728 [0.643,0.780] | 0.781 [0.708,0.843] | 0.634 [0.583,0.732] |
| TIA Centre | 0.747 [0.693,0.790] | 0.837 [0.692,0.857] | 0.677 [0.625,0.759] | 0.808 [0.683,0.837] | 0.667 [0.578,0.768] |
| Tribvn Healthcare | 0.736 [0.670,0.792] | 0.848 [0.731,0.875] | 0.631 [0.608,0.768] | 0.795 [0.633,0.848] | 0.557 [0.498,0.712] |
| CGV | 0.724 [0.657,0.779] | 0.829 [0.750,0.867] | 0.643 [0.547,0.728] | 0.675 [0.639,0.836] | 0.557 [0.521,0.697] |
| XidianU-OUC | 0.707 [0.633,0.768] | 0.800 [0.703,0.863] | 0.655 [0.528,0.696] | 0.795 [0.586,0.814] | 0.673 [0.487,0.696] |
| IAMLAB | 0.706 [0.650,0.748] | 0.695 [0.646,0.809] | 0.721 [0.642,0.757] | 0.710 [0.496,0.824] | 0.690 [0.493,0.681] |
| No.0 | 0.701 [0.637,0.752] | 0.826 [0.652,0.837] | 0.698 [0.562,0.718] | 0.757 [0.571,0.777] | 0.632 [0.553,0.696] |
| jdex | 0.696 [0.639,0.739] | 0.782 [0.737,0.820] | 0.682 [0.575,0.751] | 0.667 [0.542,0.741] | 0.430 [0.459,0.688] |
| Leeds | 0.686 [0.620,0.737] | 0.774 [0.624,0.795] | 0.549 [0.503,0.699] | 0.696 [0.594,0.786] | 0.632 [0.547,0.742] |
| PixelPath-AI | 0.676 [0.615,0.723] | 0.620 [0.636,0.788] | 0.610 [0.542,0.721] | 0.751 [0.607,0.816] | 0.576 [0.453,0.667] |
| SK | 0.671 [0.607,0.716] | 0.630 [0.636,0.783] | 0.644 [0.563,0.730] | 0.582 [0.519,0.714] | 0.637 [0.464,0.693] |
| ML | 0.632 [0.536,0.713] | 0.738 [0.586,0.821] | 0.375 [0.267,0.592] | 0.755 [0.482,0.820] | 0.514 [0.459,0.674] |
F1, precision and recall scores of all participants across scanners (bootstrapping result, 20 random case draws with replacement, 10,000 repetitions). (Figure 5 of the challenge report)
Table 1: Overview of methods submitted to the MIDOG challenge
| Team name | Core method/architecture | Multi-stage | Ensemble or TTA | Aug.: Color | Aug.: Staining | Aug.: Brightness | Aug.: Contrast | Aug.: Synthesis | Domain adaptation: Staining normalization | Domain adaptation: Other | Used unlabelled domain | Used additional labels |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reference method (Wilm et al., 2022) | RetinaNet with ResNet-18 encoder | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | Domain-adversarial training | ✓ | ✗ |
| CNN baseline (Wilm et al., 2022) | RetinaNet with ResNet-18 encoder | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| AI medical (Yang et al., 2022) | SK-UNet | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | Fourier domain mixing | ✗ | ✗ | ✓ | Mitosis segmentations |
| TIA Centre (Jahanifar et al., 2022) | Efficient-UNet (candidate segmentation), Efficient-Net-B7 (candidate classification) | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | Vahadane et al. | ✗ | ✗ | Mitosis segmentations |
| Tribun Healthcare (Fick et al., 2022) | Mask-RCNN (candidate detection), ResNet-50 and DenseNet-201 (candidate classification) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | CycleGAN | ✗ | ✗ | ✓ | Mitosis segmentations |
| CGV (Chung et al., 2022) | RetinaNet with ResNet-101 encoder | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | StarGAN | ✗ | ✗ | ✓ | ✗ |
| XidianU-OUC (Liang et al., 2022) | DetectorRS (candidate detection), ensemble of 5 models for candidate classification | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | Macenko et al. | ✗ | ✗ | ✗ |
| IAMLAB (Razavi et al., 2022) | Cascade R-CNN with ResNet-101 encoder | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | Macenko et al. | ✗ | ✗ | ✗ |
| No.0 (Long et al., 2022) | Cascade R-CNN with ResNet-50 encoder | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | Domain-adversarial training with PatchGAN | ✓ | ✗ |
| jdex (Dexl et al., 2022) | RetinaNet with Efficient-Net-B0 encoder | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Leeds (Breen et al., 2022) | UNet with ResNet-152 encoder | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| PixelPath-AI (Nateghi and Pourakpour, 2022) | Faster-RCNN (candidate detection), Efficient-Net-B0 (candidate classification) | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| SK (Kondo, 2022) | Thresholding of the blue ratio image (candidate detection), ResNet (candidate classification) | ✓ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | Domain-adversarial training | ✓ | ✗ |
| ML (Lafarge and Koelzer, 2022) | Rotation-invariant CNN with 7 trainable layers | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
The participants' papers are listed on the Publications page.
Prizes
Best contribution (as of F1 score on test set) will receive 500€, second 300€ and third 200€. (grand-challenge.org)
The challenge organizers would like to thank Siemens Healthineers for donating the monetary prizes of the challenge, who have been awarded to the top three participants. (challenge report, Acknowledgement)