Evaluating a Web-based Adult ADHD Toolkit for Primary Care Clinicians

Natalia Y. Loskutova, MD, PhD; Cory B. Lutgen, BS; Elisabeth F. Callen, PhD, GStat; Melissa K. Filippi, PhD, MPH; Elise A. Robertson, MA


J Am Board Fam Med. 2021;34(4):741-752. 

In This Article


This article describes the development and evaluation of the AAFP Adult ADHD Toolkit, as well as implementation and evaluation results regarding provider outcomes on use, knowledge, confidence, and perceived value of the Toolkit. Recent studies highlight the importance of stakeholder engagement in the research process,[6] including resource development.[6–9] Stakeholder input is more likely to produce materials that are pertinent and tailored to intended audiences versus those that do not incorporate such perspectives.[7,8,10,11] For this study, we employed an eclectic expert panel (family physicians, a pediatrician, a psychiatric clinical pharmacist, and a patient) to develop the Toolkit. All members held a unique understanding of the topic, and they contributed to clinically relevant detail and/or the appropriateness of the material for those adults who have ADHD. Our stakeholder engagement approach ensured the creation of a comprehensive adult ADHD resource by using a thorough and iterative process of content material creation and review via individual and group input.

The results of our study indicated that the Toolkit shows promise in improving PCPs' knowledge and competence related to adult ADHD even over a relatively short duration. This is of particular importance as many studies have underlined the lack of adequate education and training PCPs' receive in adult ADHD.[12–14] This insufficient education and training have yielded lower PCP awareness, understanding and knowledge, and lack of confidence in diagnosing, treating, and managing adult ADHD.[15–17] The improvements observed in several other challenging areas, particularly in domains related to managing ADHD with coexisting conditions including substance use disorders, are especially encouraging. The knowledge and competence improvements are not surprising given that those who used the Toolkit, accessed it most in the first month post-introduction. In addition, the majority used it for self-education and seeking guidance on best practices. Interestingly, knowledge and confidence improved in both groups, including those who reported they did not use the Toolkit in their practice. Training on and access to the Toolkit were provided to all, potentially boosting the practitioners' confidence. Web analytics data also indicated that the participants may have visited the Toolkit initially without actually using it in practice. In assessing knowledge levels, we relied on self-reports and cannot confirm to what extent these levels would correlate with any objective measures such as test scores in either Users or Non-Users. It is unknown, however, whether once incorporated into care, the Toolkit results in optimal adult ADHD patient care. The "knowledge to practice" translation of this resource will need to be assessed in future studies.

Specific factors related to uptake of practice and provider resources are not fully understood. In general, the uptake of practice and provider resources has been slow, due mainly to the absence of available time, which has caused issues with being able to effectively treat a patient.[18,19] Uptake can be enhanced when combining multiple strategies to grab the user's attention.[20] More recent studies have pointed to the effectiveness of providing an online resource, such as a toolkit, to provide the necessary resources for providers, but also note that users are looking for specific materials and any online resource has to be usable for a wide audience.[21,22] One of the hypotheses in this study was related to the resident status and the difference in how they might use the Toolkit. In total, 34 residents participated in this study which contributes to the relatively young mean age of our study group. The residents, however, did not use the Toolkit any differently than their more experienced nonresident colleagues. Our study shows that the use of the Toolkit was associated only with whether the providers were seeing patients with ADHD during the evaluation period. Having patients with ADHD may have determined the need for additional resources such as this toolkit. PCPs on average reported seeing about 1 to 3 patients a week. Interestingly, the number of patients with adult ADHD seen in a week was not associated with the use of the Toolkit and the use did not increase with having more patients. On one hand, this indicates that toolkits, such as this one, may be relevant to those providers who see and treat patients with a target condition and may not be universally utilized by all PCPs. On the other hand, the evaluation period was short, and it may be coincidental that some providers did not encounter patients with ADHD over several weeks of this project's duration. About two-thirds of participants reported seeing no patients with adult ADHD over the course the study. In addition to a short evaluation period, adult ADHD is often not detected and misdiagnosed in primary care. Low competence in recognizing and screening for ADHD in adults among the study participants could also potentially lead to underreporting of encountering patients with adult ADHD in some PCPs. Future studies of longer duration including patient outcomes related to screening and diagnostic assessment are needed to confirm our findings.

The uncertainty around the effectiveness of toolkits concerning improved clinical outcomes has been a question of debate in the literature. Evaluated toolkits have generally been found to have high user satisfaction[23] and effective knowledge translation[24] for clinical users, and similar studies that evaluate self-reported provider knowledge and confidence have found toolkits to have a positive effect.[24] In addition, toolkits are a simple, flexible, and expedient knowledge transfer intervention making them a pragmatic tool for implementation with many organizations continuously investing resources in their development and dissemination.[25] Our study showed that the AAFP Adult ADHD Toolkit addressed most of the providers' needs they stated, increase levels of knowledge and competence, and uncovered a need for additional education, training, and resources potentially beneficial for PCPs. Future studies should assess whether the Toolkit or its components are positively associated with clinical, safety, and care quality outcomes.

The implementation process shows that PCP participants had a sustained interest in this topic as an overwhelming majority completed the midpoint and end of study surveys. Though Toolkit use fell drastically after the first month, the portion of study participants continued visiting the Toolkit and downloading its resources through the end of the evaluation period. These findings suggest the content received in the first month of the study satisfied PCPs' most pressing needs for diagnosis and treatment. Even though we have not explored the actual reasons for the use patterns we observed, the lower use after the initial first month of the study may be related to a decreased need to visit the Toolkit after downloading the relevant information. Yet, it may prove beneficial to release Toolkit information at various times or add an interactive portion of the Toolkit to foster continued use.

Results of our study need to be interpreted with caution as they may not be generalizable to all practice types and contexts. This toolkit implementation and evaluation were conducted in only 6 practices. These 6 practices, although representing typical primary care practices, include several large academic practices that are likely to be more interested in adult ADHD. Further work is needed to formally test the implementation of the toolkit in a variety of practices. The Toolkit development may have also not fully considered the perspectives of internal medicine or clinical psychology as these specialties were not represented on the expert panel. These specialists however participated in the Toolkit evaluation, so their input is included in this work. This evaluation did not assess effectiveness for improving patient, clinical, and safety outcomes or quantify cost and resources for implementation. Due to the short implementation period, it was not feasible to assess the long-term effects or sustainability of Toolkit use. We did not explore the reasons why many providers reported seeing no patients with adult ADHD over the study period or why their weekly reports on patient encounters during the study did not align with the average numbers of adult patients with ADHD they reported before the study. We did not compare the Toolkit to other quality improvement methods. Furthermore, observations from the development phase identified evidence gaps, including resources on women, older individuals, breastfeeding, and medication management, for example, medication tapering, chronic use of stimulants, and medication holidays, among others. Therefore, our analysis does not include these particular topics due to a lack of evidence-based data. These information gaps would serve as additional points of information for Toolkit enhancements and upgrades. As far as the next steps, we plan to make the Toolkit available for PCPs and primary care practices so content materials can be distributed widely and improve on adult ADHD diagnosis and patient care needs.