Methodology for Developing Quality Indicators for the Care of Older People in the Emergency Department

Melinda Martin-Khan; Ellen Burkett; Linda Schnitker; Richard N Jones; Leonard C Gray


BMC Emerg Med. 2013;13(23) 

In This Article

Phase 2: Field Study


The purpose of the field study is to test the feasibility and usefulness of each of the preliminary indicators suggested in Phase 1. This will be achieved by collecting data from a representative sample of older patients presenting for emergency department care. The assessment of potential QIs will include a complex analytic process that involves risk adjustment.


The study will be a multi-centre prospective observational cohort study of the validity and feasibility of the preliminary QIs developed by the study team, including any previously published relevant QIs.[35]

Working from the defined preliminary indicators recorded in each of the working manuals, a matrix of data items and data collection methods will be created which ensures that, for each potential QI, the relevant data items have been identified and a collection method found (EB, LS, MMK). Based on the data matrix, the data collection tools will be designed. Wherever possible, existing, validated, tools will be used for the data collection. If a tool cannot be identified to collect specific data, then a data collection tool will be designed. The tool will be tested for feasibility at several sites at the beginning of the project and feedback from the research nurses will enable refinement of the tool.

If feasible, data to score all QIs will be collected. In addition, if feasible, additional data to support the internal validity of the QI will also be collected. It is anticipated that data collection methods will include:

  1. Site visit

  2. Direct patient assessment

  3. Review of the patient chart

  4. Extraction of data from the electronic information system in ED

  5. Phone follow-up at consecutive time intervals (7 and 28 days) with patient

  6. State held ED and hospital episode data.

QIs will only be excluded at this point if a novel data collection method is identified (separate from the above list), and the cost of additional data collection is prohibitive. Any indicators excluded at this point will be recorded, with the justification, in the Excluded Indicators Manual.

Sample Size

The sample size is determined in two ways using simulation methods resulting in a required sample of 480 participants. This planned sample size will have 77% power to detect reliability coefficients within an acceptable level of precision (estimated correlation among raters coefficient greater than 0.35 when the true value is 0.6 and the QI base rate is 50%). Given these parameters, for the classification analysis, we will be able to correctly classify units as poor (proportion of patients flagging a QI is less than the observed 20th percentile across facilities and the true quality score for the facility is below the 20th percentile) with an overall 83% accuracy. The empirical c-statistic for this classification (proportion of facilities with true performance in the lowest 20% that have observed quality scores in the lowest 20%) is 0.98.

Participants or Study Groups

The minimum number required from each site will be 60 cases. We will aim to recruit 80 cases across eight units to allow for incomplete data. This will enable a final sample size of at least 480 cases. Site selection will be influenced by case-mix and ability to recruit adequate patient numbers, with the final group allowing for representation of district, metropolitan and tertiary Emergency Departments.

Inclusion Criteria

All patients aged 70 years and older presenting to study site emergency departments during the study period will be considered eligible for enrolment.

Exclusion Criteria

  1. Patients who have presented to the ED and have completed triage 2 or more hours prior to the Research Nurse being available to approach them for consent

  2. Patients presenting to ED with acute illness of such severity that prevents staff from gaining consent (either from the individual or their caregiver)

  3. Patients returning to the ED, after already being consented for participation at the initial ED visit. Aspects relevant to their return to ED will be identified via the phone follow up process, from chart audit and from State held data on ED visits

  4. For non-English speaking patients, staff will attempt to identify a suitable interpreter to seek consent for participation – if no interpreter can be found in a suitable time-frame then the patient will be excluded

  5. Patients need to be able to participate in the planned phone follow-up – therefore itinerant patients or those with no telephone will be excluded from participation in the trial. Note: At recruitment, additional phone contact numbers will be sought from the patient, this may include family members or neighbours, to minimise loss to follow-up

  6. Patients presenting to the ED outside of the recruitment hours (Mon-Fri 8 hours per day)

A record will be kept detailing the reason for each exclusion. Demographic details (age, gender, residential location – community or residential care, triage category) will also be collected to identify if the excluded population is different from the sample population to a level reaching statistical significance.

Data Collection Tools

Site Visit The site visit tool will be designed to focus on environmental factors relating to clinical care, and structural processes (such as policies and procedures, training and staff allocation). The survey strategy will assess physical layout, equipment, staffing levels and policies and procedures pertinent to the ED management of geriatric patients.

Each site visit will take two days and involve discussions with a range of staff attached to the ED. This will include: emergency staff specialists; nursing unit managers; pharmacists; allied health and other members from the multidisciplinary team; quality managers; and ED clerical staff. At the completion of the site visit, the data will be reviewed and a list of incomplete questions will be forwarded to the site coordinator to enable the site to provide additional information that may not have been available at the time of the physical site visit. For each structural QI, data will be sought (if relevant) to support the validation of the QI at the level of policy, protocol, processes implemented and audit (regular review to identify if policy and/or processes are adhered to).

  • Prospective data collection: Several formal tools will be utilised to collect data from patients while they are in the ED. This will include a tool which provide an overview of the health status of an older person both prior to the onset of the acute episode and during the ED visit (interRAI ED Assessment[36]and selected items from interRAI Acute Care[37,38]). Input from the expert panel will be sought prior to finalisation of included tools. Final battery of assessments is anticipated to include the following standardized assessment or screening tools:

  • Cognition (Six Item Screener (SIS);[39] Orientation-Memory-Concentration (OMC) test;[40] RUDAS[41]); interRAI Cognitive Performance Scale(CPS)[42])

  • Delirium (Confusion Assessment Method (CAM);[43] interRAI delirium screener[44])

  • Pressure ulcers (The Waterlow Scale[45])

  • Pain (0–10 Numeric Rating (Pain Intensity) Scale (NRS),[46–48] Pain Assessment in Advanced Dementia Scale (PAINAD)[49]

  • Falls risk (Falls Risk for Older People in the Community (FROP-COM) Screening Tool[50])

  • Discharge risk (Identification of Seniors at Risk Tool (ISAR),[22,51,52] Triage Risk Stratification Tool (TRST),[17,53,54] InterRAI risk screeners,[55] Rowland[56])

Any additional data items (such as medication data, demographic data) will be added to the data collection tools as required.

Phone Follow-up

A phone follow up data collection form will be created to collect data on:

  • Adverse events following the ED episode

  • Additional ED or acute care hospital admissions

  • Time spent in residential care (respite or newly admitted as permanent resident)

  • Pain and medication management following the ED episode

  • Patient satisfaction

  • Patient perception of clinical decision making and privacy

Phone follow up will occur at 7 and 28 days but data will be combined to provide a summary of the total 28-day period. Two phone calls are scheduled to ensure continuity of contact with the patient and to assist in more reliable recall of information over shorter periods of time.

Emergency Department Information System (EDIS) Data Extraction

A data collection form will be created to identify key information stored on the system relevant to the patient and to compare this with data collection from the patient. The EDIS data will also be used to provide general demographic data relating to patients both included and excluded (gender, age, triage category, residential setting).

Patient Medical Record

This record will vary between institutions, being either electronic, paper based or a combination of both. A chart review tool will be designed which focuses on abstraction of data, and minimises the need for 'interpretation' of data during the audit process. Where possible, existing chart abstraction tools will be utilized. The final chart review tool will undergo preliminary pilot testing.

Data Custodian Information on ED Episodes and Acute Care Admissions

Data on the index ED episode, subsequent acute care admission and any additional hospital interactions in the 28 days post ED departure will be sought from the data custodian in each State. International Classification of Diseases (10th revision)[57] codes, primary diagnoses, length of stay and classification of care for each episode of care during the study time period will be requested.

Each data collection set (comprises all the data collection sheets for each phase of data collection) will be matched against the original data matrix to ensure that all required variables are being collected. A database for data entry will be created. Each variable item, coded to match the data collection sheet and carrying a unique variable name, will be recorded in the manual alongside each QI in preparation for analysis of the data set.

Research Staff

A registered nurse with geriatric assessment expertise(site nurse) will be employed for prospective data collection, including the phone follow-up, at each site. All staff will attend the Centre for Research in Geriatric Medicine (CRGM) for comprehensive training prior to data collection commencing. Each enrolled patient will be assessed by the trained research nurse utilising the compilation of data collection tools – this will allow for a comparison of data collected by the research nurse, who will complete chart reviews, and the site nurses, thus allowing an evaluation of the reliability of QI information obtained by chart audit (through triangulation of data). The patient will otherwise undergo usual ED assessment and management.

Two research staff, with nursing backgrounds will be trained to complete the site visits. One site visit will be completed jointly, but scored separately to test the data collection tool with respect to inter-rater reliability. All remaining site visits will be visited by one of the two research staff.

Research nursing staff will be trained to complete the chart review. The data will be collected in a retrospective fashion by trained chart/database abstractors using a standardized chart abstraction protocol – these abstractors will be blinded to the site nurse assessment. The training will include the protocol, supervised practice charts and independent chart review followed by comparison with trainer review. 5% of charts will be co-reviewed to ensure a kappa of > 0.7, which by convention suggests excellent inter-rater reliability.[58]

Staff carrying out the data collection will be blinded to the individual QIs. All data items, regardless of the data collection method (prospective, chart review, site visit) will be standalone items and not be grouped or identified in the data collection sheet as linked to an individual QI.

Data Collection

The research nurse at each site will identify eligible patients at the beginning of each shift using the EDIS. All eligible patients will be approached in consecutive order. If a patient becomes ineligible or is excluded, general demographic information will be recorded, along with the reason for ineligibility. For eligible patients, the research nurse will explain the purpose of the study, the range of questions that will be asked and the anticipated duration of the patient's involvement and seek written consent from the patient or a nominated secondary decision maker for participation.

The research nurse will confirm general contact and demographic information with the patient. The initial data collection questions will focus on the patient's current condition or situation, and include items relating to cognition, delirium, pain, medications, skin integrity and continence (these questions relate to aspects of health that may change before and during the ED episode). A second series of questions will be related to the patient's situation prior to the onset of the acute medical condition, the reason for attending the ED, and arrangements for additional care following the ED episode (capacity to get home, additional nursing care, etc.).

If the patient is in the ED for 3 or more hours, the research nurse will return to the patient and repeat the section of the initial data collection questions which focused on the patient's condition (cognition, delirium, etc.).

Following the patient's departure from the ED, the research nurse will identify the discharge medications, discharge location, length of stay in the Emergency Department and other general details.

Seven days following the departure from the ED, the research nurse will contact the patient and complete the phone follow up. At 28 days, the phone follow up will be repeated with a small section of repeated questions, which relate to any adverse events in the preceding days.

The site visit is organised separately from the prospective data collection and will occur throughout the data collection period.

The chart reviews will be completed following the end of the prospective data collection period. They will commence no sooner than two months after data collection had been completed. This will enable all relevant information to be filed in the chart. All patient medical records will be recalled and the chart abstractors will review each chart using a pilot tested audit tool.

Finally, no sooner than six months after the end of the prospective data collection, the data custodian will be contacted to request the relevant information regarding the index ED episode and any other hospital events up to and including 28 days post ED departure. The time lapse is to ensure that all data has been received from the State's hospitals.

Data Compilation

A recruitment database will be completed by the site nurse and forwarded, at regular intervals, to CRGM for review. The recruitment database will hold the general demographic information and the unique research identifies for each consenting patient.

All data, assembled by the research nurse for prospective data collection, will be de-identified and forwarded to CRGM for data entry into an electronic database. Each file will be reviewed by one researcher for completeness prior to data entry. Any issues will be reconciled by request to the research nurse at each site.

A second researcher will review all site visit data and request any missing information from each site coordinator. The data will then be entered into an electronic database ready for analysis.

Finally, the chart abstraction data collection tools will be forwarded to CRGM, identified only by the unique research ID, reviewed by one researcher for completeness and entered into an electronic database.

A separate database will be established for each site. When all data is entered, and checked, the databases will be combined to establish one complete database ready for analysis.