Methodology for Developing Quality Indicators for the Care of Older People in the Emergency Department

Melinda Martin-Khan; Ellen Burkett; Linda Schnitker; Richard N Jones; Leonard C Gray


BMC Emerg Med. 2013;13(23) 

In This Article

Phase 3: Expert Panel and Voting Rounds


The purpose of the final expert panel is to enable panel to review the preliminary indicators alongside the data from the field study (phase 2) and revise or exclude indicators prior to the voting round. The voting rounds will culminate in the assembly of a final QI set that will reflect quality of care in terms of structure, process and outcomes.


This phase will comprise the latter stages of analysis of results of the field study, preparation of reports to inform the expert panel, a two day seminar to consider the findings of the field study and assembly of the final QI set with associated recommendations.

A formal report will be prepared for general scrutiny in addition to publication for the peer-reviewed literature. A formal procedure for selection of the final QI set will follow the expert panel deliberations, similar to that used in assembly of the Assessing Care Of Vulnerable Elders (ACOVE) indicators.[59] This process involves two rounds of anonymous ratings on a risk-benefit scale with a teleconference group discussion occurring between rounds.[60,61]

Data Analysis

Primary analysis will be to evaluate the new QIs. The QIs will be adjusted for ascertainment and selection bias through risk adjustment procedures.[58] The determination of appropriate case-mix and risk adjustment procedures will involve simple bi-variable descriptive statistics (correlations, mean differences). Good candidates for adjustment will be included as matching criteria in the QI adjustment process. The QI adjustment method will use a procedure that has the advantage of being quasi-parametric, involving matching individual patients in target EDs to randomly selected patients from other EDs. This counterfactual contrast will include a re-sampling procedure and allow QIs to be expressed as odds ratios or expected proportions given an overall average rate and an empirically based replication (i.e. confidence) interval. Relative to extant methods of risk adjustment this approach is relatively simple, can be implemented in clinical populations of small size and represents as perfect as possible adjustment for differences in patient mix across clinical settings.

The reliability of QI scores will be evaluated by multiple bootstrapped split-half correlations of patient samples and time-to-time correlations of repeated QI scores. This is a unit-level analysis, where for each ED we will use a bootstrapping data augmentation approach to generate 20 random half samples of patients.

Consideration of the issues specific to patients with cognitive impairment, nursing home residents and those patients requiring palliative care will result in an additional analysis of QI data to identify whether any QIs are specifically significant for these sub-groups.

Comparisons with SAEM QIs will use standard methods for comparing correlation coefficients for the contrasting reliability coefficients, and cross tabulations of tertiles of QIs in similar domains for the validity assessment.


Following the final expert panel, the indicators will be presented to the expert panel in a summary document. In the document, each indicator will be described in relation to the agreed name, denominator, numerator and exclusion criteria. A short summary of relevant evidence supporting the indicator and a précis of the panels' discussion in relation to how well the indicator aligned with the selection criteria, will be included. There will be graphical representation of the field study data, including prevalence (raw scores) of the trigger rates, and percentage scores.

There will be a formal voting process, involving two voting rounds, following the RAND-UCLA Appropriateness Method.[62,63] The panel will be asked to rate each indicator with a score from one to nine based on its validity when considered in relation to the selection criteria. The selection criteria include:

  • Criteria 1: Quality of Care indicator - Adequate scientific evidence or professional consensus supported a link between the process specified by the indicator and a health benefit to the patient; an ED with high rates of adherence to the indicator would be considered a higher-quality provider

  • Criteria 2: Measurement accuracy - Ideally the indicator would be measured using a gold standard measure or a measure with proven robust attributes for the measured population when administered appropriately; the indicator measures what it is meant to measure

  • Criteria 3: Provider Control - An ED influences a majority of the factors that determine the outcome of the indicator (relevant to the inpatient episode of care)

  • Criteria 4: Generalisability – The indicator is relevant to a high proportion of the targeted population

  • Criteria 5: Responsiveness - The indicator is responsive to changes over time; that is, it will be possible to identify and measure the impact of interventions designed to improve care. (i.e. evidence that there are interventions which can lead to improvement in care)

  • Criteria 6: Event Rate - Occurs frequently and is of sufficient significance that monitoring should occur

Voting sheets will be returned to CRGM, where they will be collated. A second round of voting sheets will be distributed to the panel. Each individualised voting sheet will include: the de-identified votes of the panel (i.e. how many panel members voted '1', how many voted '2', etc.) for each indicator; the actual vote of the panel member from round one; summary of the panel votes including the median vote; the mean standard deviation from the median; presence of agreement (or disagreement) in relation to that indicator; result of the round one vote (indicator valid, undecided or invalid). Panel summary statistics will be calculated after removing the highest and lowest vote for each indicator (i.e. the most extreme votes).

Agreement is decided by calculating the Interpercentile Range Adjusted for Symmetry (IPRAS) and the Interpercentile Range (IPR).[62] If the IPRAS is larger than the IPR then there is agreement in the panel on a particular indicator. The indicator is valid if the median score is between seven and nine, and the panel are in agreement. A median with a decimal of 0.5 or higher is rounded up.

As this voting process is a consensus method, there will be a teleconference to discuss the voting round. The focus of discussion is on indicators where there was disagreement in round one. In some instances, disagreement occurs because of a misunderstanding or lack of clarity in the definition. This discussion allows the opportunity to clarify the definition such that it improves the usefulness of the final indicator. In some instances, the disagreement occurs because of a difference in opinion. Given the multi-disciplinary nature of the panel, this teleconference enables one last opportunity for evidence to be highlighted in support of a point of view.

The panel then vote for a second time on all indicators. They can repeat their vote or move their vote up or down the scale to strengthen the impact of their opinion. All indicators identified as valid in this second round of voting, will be incorporated in the final set. If there is one care domain where no valid indicators are identified, but there are indicators where the vote is 'undecided' (median score was 4–6 or there was disagreement (IPRAS less than or equal to IPR), then the undecided indicator with the highest median (taking into account decimal places) will be included in the final indicator set.

Integration of Findings

Dissemination of findings will be undertaken by publication in peer reviewed Emergency medicine and Medical Administration Journals of:

  1. Scientific reviews of the literature undertaken to allow optimal evidence-base for development of robust QIs

  2. A final recommended QI set for care of elderly in the ED

Following the above project, the finalised set of QIs will be subjected to a more widespread validation study. Results of this study will be a validated set of QIs for care of older persons in ED – these will be presented to key Australian and international Emergency Medicine Colleges and Societies and to national and international accrediting boards for consideration of ratification. In addition, presentations are planned at national and international conferences to communicate results to attendees. Finally, the use of these QIs by clinical investigators as outcome measures, supplementary to their project specific measures, will be encouraged by the research team.

Given that existing QIs will be compared to indicators developed in this project, stakeholders will be empowered to choose those indicators that will most optimally fulfil their specific goals.