Accuracy and Efficiency of Deep-Learning–Based Automation of Dual Stain Cytology in Cervical Cancer Screening

Nicolas Wentzensen, MD; Bernd Lahrmann, PhD; Megan A. Clarke, PhD; Walter Kinney, MD; Diane Tokugawa, MD; Nancy Poitras, BS; Alex Locke, MD; Liam Bartels, BS; Alexandra Krauthoff, BS; Joan Walker, MD; Rosemary Zuna, MD; Kiranjit K. Grewal, MS; Patricia E. Goldhoff, MD; Julie D. Kingery, MD; Philip E. Castle, PhD; Mark Schiffman, MD; Thomas S. Lorey, MD; Niels Grabe, PhD

Disclosures

J Natl Cancer Inst. 2021;113(1):72-79. 

In This Article

Results

Automated Detection of DS-positive Cells in Colposcopy and Anoscopy Populations

We developed a deep-learning algorithm for automated detection of DS-positive cells on ThinPrep slides from 2 referral populations (Biopsy Study and ACSS), including 212 training slides with 1186 DS-positive and 1485 DS-negative tiles (Figure 1). We evaluated the algorithm in independent validation slides from both studies (Figure 1). In both studies, we observed an increase in the number of DS-positive cells by increasing severity of cytology and histology, with higher absolute DS-positive cell numbers in ACSS (P < .001 for all comparison; Supplementary Figure 2, available online).

In the Biopsy Study validation set with 53 CIN3+, the AUC for detecting CIN3+ based on automated DS using CNN4 was 0.74 (Figure 2). At a cutoff of 3 DS-positive cells, the CNN4 algorithm had marginally lower positivity (58% vs 63%, respectively, P = .06) with comparable sensitivity (P = 1.0) and marginally higher specificity compared with manual DS (40.6% vs 45.7%, respectively, P = .07) (Table 1).

Figure 2.

Receiver operating curve characteristics analysis of number of dual stain (DS)-positive cells detected by CYTOREADER for detection of cervical precancer in the Biopsy Study and anal precancer in the Anal Cancer Screening Study. AUC = area under the curve; AIN2+ = anal intraepithelial neoplasia grade 2 or worse; CIN3+ = cervical intraepithelial neoplasia grade 3 or worse.

In the ACSS validation set with 69 AIN2+, the AUC for detecting AIN2+ based on automated evaluation of DS slides with CNN4 was 0.77 (Figure 2). At a cutoff of 3 DS-positive cells, the positivity of the CNN4 algorithm was lower (63% vs 71%, respectively, P = .001) with comparable sensitivity (P = 1.0) and higher specificity compared with manual DS (36.1% vs 46.1%, respectively, P = .001) (Table 1).

Automated Detection of DS-positive Cells in an HPV Screening Population

We developed the deep-learning algorithm for SurePath slides using a training set of 238 slides from the Kaiser study with 8215 DS-positive and 9739 DS-negative tiles and applied it in an independent validation set of slides from 3095 women. We observed an increase of DS-positive cells with increasing severity of cytology and histology (Supplementary Figure 3, available online).

In the Kaiser validation study including 218 CIN3+, the AUC for detecting CIN3+ based on automated evaluation of DS slides was 0.82 (Figure 3). At a cutoff of 2 cells, the positivity of the algorithm was statistically significantly lower (42% vs 50%, respectively, P < .001) with equal sensitivity (P = .4) but statistically significantly higher specificity (61.5% vs 52.6%, respectively, P < .001) compared with the manual DS. At a cutoff of 100 cells, accuracy approached HSIL cytology that allows for immediate treatment according to current management guidelines (Figure 3). Automated DS provided better risk stratification compared with Pap cytology and manual DS (Figure 4): more women were reassured of a lower risk compared with the other strategies (58% for automated DS vs 50% for manual DS and 40% for cytology), and risk among positives was higher.

Figure 3.

Receiver operating curve characteristics analysis of number of dual stain (DS)-positive cells detected by CYTOREADER for detection of cervical precancer in a human papillomavirus screening population in Kaiser Permanente Northern California. ASC-US = Atypical Squamous Cells of Undetermined Significance; AUC = area under the curve; HSIL = high-grade squamous intraepithelial lesions.

Figure 4.

Absolute risk of precancer for Pap cytology, manual dual stain (DS), and automated DS. ASCUS+ = positive for Atypical Squamous Cells of Undetermined Significance or greater cytology results. The dotted lines show clinical action risk thresholds for colposcopy referral (4% risk) and immediate treatment (50% risk).

Clinical Efficiency of Automated DS Evaluation

We compared the clinical efficiency of Pap cytology, manual DS, and automated DS at 2 cutoffs (2 or more cells and 1 or more cells) for triage of HPV-positive women (Table 2). All DS strategies achieved equal or better sensitivity for detection of CIN3+ compared with Pap cytology while reducing unnecessary colposcopic referrals. Automated DS reduced overall referral to colposcopy by one-third for the primary automated cutoff of 2 cells (41.9% for automated DS vs 60.1% for cytology, P < .001). Automated DS at a cutoff of 2 or more cells had similar sensitivity but statistically significantly higher specificity compared with manual DS evaluation (61.5% vs 52.6%, P < .001). Automated DS detection at a cutoff of 1 or more cells achieved the highest sensitivity of all strategies, with statistically significantly higher specificity and lower colposcopy referral compared with Pap cytology. Automated DS at a cutoff of 2 or more cells had the most favorable ratio of colposcopies per CIN3+ detected compared with the least favorable ratio for the current standard, Pap cytology (6.8 vs 9.9, respectively). Extrapolating this to the full Kaiser screening population, out of 300 000 women screened annually, approximately 30 000 would test HPV-positive and more than 18 000 would be referred to colposcopy using the current approach with Pap cytology, while only 12 570 would be referred to colposcopy using automated DS. We also estimated the performance of automated DS in a fully vaccinated population. Similar to the overall evaluation, automated DS showed equal sensitivity and lower colposcopy referral compared with Pap cytology with even higher specificity (Supplementary Table 1, available online).

Comments

3090D553-9492-4563-8681-AD288FA52ACE

processing....