Evaluating a Web-based Adult ADHD Toolkit for Primary Care Clinicians

Natalia Y. Loskutova, MD, PhD; Cory B. Lutgen, BS; Elisabeth F. Callen, PhD, GStat; Melissa K. Filippi, PhD, MPH; Elise A. Robertson, MA


J Am Board Fam Med. 2021;34(4):741-752. 

In This Article


This is a single-arm repeated measures intervention study conducted in 6 practices with 97 providers from November 1, 2019, through March 31, 2020. The study evaluated the newly developed toolkit for adult ADHD detection and management in primary care. The AAFP Institutional Review Board approved this study. All project participants provided informed consent at the onset of their participation.

Toolkit Development Process

The toolkit content was created through a comprehensive research review in conjunction with expert panel input. The project team reviewed prior studies and published literature, identified existing best practice guidelines, and performed a comprehensive audit of publicly available tools and resources. Simultaneously, a panel of adult ADHD experts was convened to provide individual and group input. Five family medicine physicians, 1 pediatrician, 1 psychiatric clinical pharmacist, and 1 adult patient with ADHD comprised the panel. The panel members were identified through the AAFP National Research Network and selected based on relevant expertise among individuals who volunteered. Panel members' input consisted of individual toolkit content review, discussions in 2 1.5-hour moderated virtual group meetings and a 1-day in-person meeting. For this project, panelists reviewed, ranked, selected, and organized the materials in an iterative review and development process. Themes and content from individually completed worksheets and documented panel discussions informed the toolkit's resources, materials, structure, assembly, and format. The resulting web-based AAFP Adult ADHD Toolkit (i.e., Toolkit) includes the contents shown in Figure 1. The full Toolkit is available free of charge to any user at https://www.aafp.org/patient-care/public-health/adhd-toolkit.html.

Figure 1.

AAFP Adult ADHD Toolkit Content Overview: Sections and Sub-sections.

Toolkit Evaluation in Real-world Primary Care Setting

We evaluated the Toolkit in a practice-based setting using data from provider surveys and web analytics. The AAFP NRN recruited primary care practices that fit the following criteria: use the Toolkit for at least 17 weeks and complete a series of surveys developed by the research team. A purposeful sampling method was used to include participating practices for an equal distribution of small private practices, residency practices, and large health care systems. All providers from these practices were invited to participate. All participating providers gave informed consent and completed a baseline survey. The baseline survey assessed participant demographics, knowledge, confidence, and needs around various aspects of adult ADHD care. Practices received one-on-one training in the use of the Toolkit. We followed the practices for 17 weeks to monitor key aspects of Toolkit use, provider knowledge, confidence, and perceived value of the Toolkit. The research team distributed weekly provider surveys to monitor self-reported use of the Toolkit and collected web analytics data. At the midpoint (week 9) and the end of the implementation period, providers were assessed on when, how, and why they used the Toolkit materials and what parts they found most/least useful. Lastly, provider knowledge and competence were assessed using predefined domains.

Data Collection

The project team developed and administered online surveys which generated Toolkit evaluation data. The project team's IT department provided web analytics data. All surveys were distributed electronically through e-mail using Qualtrics (Provo, Utah).

Web analytics data were generated through an Adobe Analytics report for the stated evaluation period. The report included data for Visits, Unique Visitors, Page Views, and Downloads (where applicable) for each page and subpage of the Toolkit. In the context of this project, a visit is 1 individual visitor who viewed the Toolkit website and proceeded to browse. Total visits are reported as counts of all visits to the specified page, no matter how many times the same visitor may have been to the specified page during the specified time frame. Unique Visitors is the count of the number of unique individuals that visited a particular page during the specified time frame regardless of how many times they visited. Page Views is defined as the total number of views or visits to a particular page for the specified time period. Downloads indicate the number of times a specified document was opened in the non-html version and subsequently downloaded by a visitor. Only the project participants and the project staff had access to the web-based Toolkit during the evaluation period. The internal IP addresses from the project team's organization were excluded from Visits, Page Views, and Downloads data. None of these web analytics data were otherwise identifiable.

Data Analysis

Descriptive statistics (counts, percentages, and means with standard deviations) were used to report demographic, survey, and web analytics Toolkit use data as appropriate. Before the analyses, all participants were assigned to 2 subgroups: those who self-reported use of the Toolkit ("Users") and those who self-reported they did not use the Toolkit ("Non-Users") using cumulative survey data. For examining the characteristics of those who used versus did not use the Toolkit, we used binary logistic regressions. For the above-mentioned analyses, missing data were excluded on a case-by-case basis. For the assessment of the change over time in the knowledge and confidence between those who used the Toolkit and those who did not use the Toolkit, we used Mixed ANOVA. Data only from the participants who completed baseline, midpoint, and end of study surveys were included in the ANOVA analyses. For the knowledge and confidence assessments, average group scores on 1 to 5 point scale for each predefined domain were compared across 3 time points (baseline, midpoint, end of study) between the subgroups (used the Toolkit vs did not use the Toolkit). An α of 0.05 was used throughout the analysis. All analyses were conducted with SPSS 25 (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp).