Quality and Safety in Health Care Journal

Measuring outcomes in quality improvement education: success is in the eye of the beholder

Over the past decade, quality improvement (QI) has gone from a secret skill expected only among trained staff in the quality office to a core competency for all health professionals.1–3 This expectation has generated new curricula which have introduced QI to a new generation of learners, but has also created some challenges for health professions educators.4–7 Identifying knowledgeable teachers, defining core content and securing time in the curriculum represent recurring issues, while emerging discussions now centre on how best to evaluate educational efforts in QI. It is here that we find ourselves at an impasse.

In this issue of BMJ Quality and Safety, O’Leary and colleagues present their 5-year experience delivering an institutionally sponsored, team-based QI training programme which included attending physicians, residents and fellows and frontline interprofessional team members. They report on its impact...

Optimising detection and prevention of prosthetic joint infections

Translation of best practices to clinical practice can be a considerably lengthy process. Reducing surgical site infections (SSIs) following primary hip and knee arthroplasties is a crucial endeavour in light of the continued rise in the number of these operations being performed and the morbidity associated with prosthetic joint infections (PJIs).1 A number of interventions have been successful in lowering SSI rates following orthopaedic procedures, with those targeting Staphylococcus aureus particularly effective given that it is the most common pathogen.2 Measures to reduce SSIs are evidence-based, relatively straightforward and cheap, yet widespread implementation remains elusive. Perioperative staphylococcal decolonisation represents a substantial cost savings opportunity given the economic burden associated with PJIs, including revision operations, rehospitalisation and prolonged antibiotic courses .3

Calderwood et al4 report on the impact of disseminating a SSI prevention bundle for hip and knee PJIs using a pre-existing...

Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key

Picture yourself in a casino sitting at the roulette table. Will you put your money on either black or red, or will you go for a riskier choice and place it on just a single number? But before you can make up your mind, the elderly, obese smoker to your right reaches for his chest and collapses. ‘What is wrong with him?’, cries his accompanying wife, looking at you in shock.

In this issue, Lawton and colleagues ask whether more experienced clinicians are better able to tolerate uncertainty and manage risks.1 Doctors working in three emergency departments (ED) read four clinical vignettes each accompanied by four quite distinct options for management, all of which might be deemed clinically acceptable. Participants used 5-point Likert scales to indicate their agreement with each of the management plans. By design, the four options offered for each vignette included two management plans...

Evolving quality improvement support strategies to improve Plan-Do-Study-Act cycle fidelity: a retrospective mixed-methods study

Background

Although widely recommended as an effective approach to quality improvement (QI), the Plan–Do–Study–Act (PDSA) cycle method can be challenging to use, and low fidelity of published accounts of the method has been reported. There is little evidence on the fidelity of PDSA cycles used by front-line teams, nor how to support and improve the method’s use. Data collected from 39 front-line improvement teams provided an opportunity to retrospectively investigate PDSA cycle use and how strategies were modified to help improve this over time.

Methods

The fidelity of 421 PDSA cycles was reviewed using a predefined framework and statistical analysis examined whether fidelity changed over three annual rounds of projects. The experiences of project teams and QI support staff were investigated through document analysis and interviews.

Results

Although modest, statistically significant improvements in PDSA fidelity occurred; however, overall fidelity remained low. Challenges to achieving greater fidelity reflected problems with understanding the PDSA methodology, intention to use and application in practice. These problems were exacerbated by assumptions made in the original QI training and support strategies: that PDSA was easy to understand; that teams would be motivated and willing to use PDSA; and that PDSA is easy to apply. QI strategies that evolved to overcome these challenges included project selection process, redesign of training, increased hands-on support and investment in training QI support staff.

Conclusion

This study identifies support strategies that may help improve PDSA cycle fidelity. It provides an approach to assess minimum standards of fidelity which can be replicated elsewhere. The findings suggest achieving high PDSA fidelity requires a gradual and negotiated process to explore different perspectives and encourage new ways of working.

Immediate and long-term effects of a team-based quality improvement training programme

Background

Although many studies of quality improvement (QI) education programmes report improvement in learners’ knowledge and confidence, the impact on learners’ future engagement in QI activities is largely unknown and few studies report project measures beyond completion of the programme.

Method

We developed the Academy for Quality and Safety Improvement (AQSI) to prepare individuals, across multiple departments and professions, to lead QI. The 7-month programme consisted of class work and team-based project work. We assessed participants’ knowledge using a multiple choice test and an adapted Quality Improvement Knowledge Assessment Test (QIKAT) before and after the programme. We evaluated participants’ postprogramme QI activity and project status using surveys at 6 and 18 months.

Results

Over 5 years, 172 individuals and 32 teams participated. Participants had higher multiple choice test (71.9±12.7 vs 79.4±13.2; p<0.001) and adapted QIKAT scores (55.7±16.3 vs 61.8±14.7; p<0.001) after the programme. The majority of participants at 6 months indicated that they had applied knowledge and skills learnt to improve quality in their clinical area (129/148; 87.2%) and to implement QI interventions (92/148; 62.2%). At 18 months, nearly half (48/101; 47.5%) had led other QI projects and many (41/101; 40.6%) had provided QI mentorship to others. Overall, 14 (43.8%) teams had positive postintervention results at AQSI completion and 20 (62.5%) had positive results at some point (ie, completion, 6 months or 18 months after AQSI).

Conclusions

A team-based QI training programme resulted in a high degree of participants’ involvement in QI activities beyond completion of the programme. A majority of team projects showed improvement in project measures, often occurring after completion of the programme.

Effectiveness of a multistate quality improvement campaign in reducing risk of surgical site infections following hip and knee arthroplasty

Background

Quality improvement (QI) campaigns appear to increase use of evidence-based practices, but their effect on health outcomes is less well studied.

Objective

To assess the effect of a multistate QI campaign (Project JOINTS, Joining Organizations IN Tackling SSIs) that used the Institute for Healthcare Improvement’s Rapid Spread Network to promote adoption of evidence-based surgical site infection (SSI) prevention practices.

Methods

We analysed rates of SSI among Medicare beneficiaries undergoing hip and knee arthroplasty during preintervention (May 2010 to April 2011) and postintervention (November 2011 to September 2013) periods in five states included in a multistate trial of the Project JOINTS campaign and five matched comparison states. We used generalised linear mixed effects models and a difference-in-differences approach to estimate changes in SSI outcomes.

Results

125 070 patients underwent hip arthroplasty in 405 hospitals in intervention states, compared with 131 787 in 525 hospitals in comparison states. 170 663 patients underwent knee arthroplasty in 397 hospitals in intervention states, compared with 196 064 in 518 hospitals in comparison states. After the campaign, patients in intervention states had a 15% lower odds of developing hip arthroplasty SSIs (OR=0.85, 95% CI 0.75 to 0.96, p=0.01) and a 12% lower odds of knee arthroplasty SSIs than patients in comparison states (OR=0.88, 95% CI 0.78 to 0.99, p=0.04).

Conclusions

A larger reduction of SSI rates following hip and knee arthroplasty was shown in intervention states than in matched control states.

Are more experienced clinicians better able to tolerate uncertainty and manage risks? A vignette study of doctors in three NHS emergency departments in England

Background

Risk aversion among junior doctors that manifests as greater intervention (ordering of tests, diagnostic procedures and so on) has been proposed as one of the possible causes for increased pressure in emergency departments (EDs). Here we tested the prediction that doctors with more experience would be more tolerant of uncertainty and therefore less risk-averse in decision making.

Methods

In this cross-sectional, vignette-based study, doctors working in three EDs were asked to complete a questionnaire measuring experience (length of service in EDs), reactions to uncertainty (Gerrity et al, 1995) and risk aversion (responses about the appropriateness of patient management decisions).

Results

Data from 90 doctors were analysed. Doctors had worked in the ED for between 5 weeks and 21 years. We found a large association between experience and risk aversion so that more experienced clinicians made less risk-averse decisions (r=0.47, p<0.001). We also found a large association between experience and reactions to uncertainty (r=–0.50, p<0.001), with more experienced doctors being much more at ease with uncertainty. Mediation analyses indicated that tolerance of uncertainty partially mediated the relationship between experience and lower risk aversion, explaining about a quarter of the effect.

Conclusion

While we might be tempted to conclude from this research that experience and the ability to tolerate uncertainty lead to positive outcomes for patients (less risk-averse management strategies and higher levels of safety netting), what we are unable to conclude from this design is that these less risk-averse strategies improve patient safety.

Validation of the Primary Care Patient Measure of Safety (PC PMOS) questionnaire

Background

The Primary Care Patient Measure of Safety (PC PMOS) is designed to capture patient feedback about the contributing factors to patient safety incidents in primary care. It required further reliability and validity testing to produce a robust tool intended to improve safety in practice.

Method

490 adult patients in nine primary care practices in Greater Manchester, UK, completed the PC PMOS. Practice staff (n = 81) completed a survey on patient safety culture to assess convergent validity. Confirmatory factor analysis (CFA) assessed the construct validity and internal reliability of the PC PMOS domains and items. A multivariate analysis of variance was conducted to assess discriminant validity, and Spearman correlation was conducted to establish test–retest reliability.

Results

Initial CFA results showed data did not fit the model well (a chi-square to df ratio (CMIN/DF) = 5.68; goodness-of-fit index (GFI) = 0.61, CFI = 0.57, SRMR = 0.13 and root mean square error of approximation (RMSEA) = 0.10). On the basis of large modification indices (>10), standardised residuals >± 2.58 and assessment of item content; 22 items were removed. This revised nine-factor model (28 items) was found to fit the data satisfactorily (CMIN/DF = 2.51; GFI = 0.87, CFI = 0.91, SRMR = 0.04 and RMSEA = 0.05). New factors demonstrated good internal reliability with average inter-item correlations ranging from 0.20 to 0.70. The PC PMOS demonstrated good discriminant validity between primary care practices (F = 2.64, df = 72, p < 0.001) and showed some association with practice staff safety score (convergent validity) but failed to reach statistical significance (r = –0.64, k = 9, p = 0.06).

Conclusion

This study led to a reliable and valid 28-item PC PMOS. It could enhance or complement current data collection methods used in primary care to identify and prevent error.

Systems consultation for opioid prescribing in primary care: a qualitative study of adaptation

Background

In order to promote guideline-concordant opioid prescribing practices, a blended implementation strategy called systems consultation was pilot tested in four primary care clinics in one US health system.

Objectives

To describe (1) how systems consultation worked during the pilot test and (2) the modifications necessary to adapt this implementation strategy to primary care.

Methods

A team of investigators conducted observations (n=24), focus groups (n=4) and interviews (n=2). The team; kept contact logs documenting all interactions with the intervention clinics and preserved all work products resulting from the intervention. Initial analysis was concurrent with data collection and findings were used to modify the intervention in real time. At the conclusion of the pilot test, a pragmatic descriptive analysis of all data was performed to explore key modifications.

Results

Time constraints, entrenched hierarchical structures and a lack of quality improvement skills among clinical staff were the main barriers to implementing systems consultation. Modifications made to address these conditions included creating a consulting team, giving change teams more direction, revising process improvement tools, supporting the use of electronic health record (EHR) functionalities and providing opportunities for shared learning among clinics.

Discussion and conclusion

With the lessons of this research in mind, our goal in future iterations of systems consultation is to give clinics a combination of clinical, organisational change and EHR expertise optimised according to their needs. We believe a streamlined process for assessing the key characteristics identified in this study can be used to develop a plan for this kind of optimisation, or tailoring, and we will be developing such a process as part of an upcoming clinical trial.

Motivating and engaging frontline providers in measuring and improving team clinical performance

Background

Performance measurement (PM) and management for quality have become ubiquitous in 21st-century healthcare. Numerous entities have independently developed measures for assessing mortality, quality of chronic-disease care, access and patient satisfaction. Consequently, measures have mushroomed; for example, the National Clearinghouse for Quality Measures houses nearly 1100 active measures.1 Despite this proliferation, those whose performance is being measured have had little input in measure development. Research consistently shows when performance measurement systems are implemented by leadership divorced of the evidence-based motivational component that induces goal commitment and facilitates behaviour change, these do not accelerate performance improvement.2 3

For example, traditional PM systems like the Healthcare Effectiveness Data and Information Set have focused on disease prevalence yet lack a conceptual model of the clinical-performance criterion domain. Outside of healthcare, performance measurement systems such as Total Quality Management (TQM), Lean/Six Sigma and Balanced Scorecards are...

Physician characteristics associated with patient experience scores: implications for adjusting public reporting of individual physician scores

Introduction

Public reporting of individual physician patient experience scores is becoming widespread on hospital websites and may be included on the Centers for Medicare and Medicaid Services Physician Compare website in the future.1 2 Moreover, physician groups can submit patient experience scores for incentive compensation through the Merit-based Incentive Payment System, and many groups are including patient experience scores in the allocation of incentive compensation.3 However, there is concern that certain physician characteristics (eg, gender, age, training) may unduly influence these scores and may merit consideration of adjustment when used in these ways. It is important to understand whether uncontrollable factors beyond the patient–physician encounter may influence patients’ scores regarding a physician. It was our hypothesis that physician factors such as age, sex and aspects of his or her medical training significantly influence patient experience scores. Therefore, our objective was to assess...

Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback

Audit and feedback (A&F) is a commonly used quality improvement (QI) approach. A Cochrane review indicates that A&F is generally effective and leads to modest improvements in professional practice but with considerable variation in the observed effects. While we have some understanding of factors that enhance the effects of A&F, further research needs to explore when A&F is most likely to be effective and how to optimise it. To do this, we need to move away from two-arm trials of A&F compared with control in favour of head-to-head trials of different ways of providing A&F. This paper describes implementation laboratories involving collaborations between healthcare organisations providing A&F at scale, and researchers, to embed head-to-head trials into routine QI programmes. This can improve effectiveness while producing generalisable knowledge about how to optimise A&F. We also describe an international meta-laboratory that aims to maximise cross-laboratory learning and facilitate coordination of A&F research.

Quality & safety in the literature: May 2019

Key points

  • In a systematic review exploring burnout in practicing physicians, marked variability in study quality and definitions of and methods used to assess burnout precluded the ability to accurately determine the prevalence of this condition. A consensus definition of burnout and uniform methods to assess it are needed. JAMA. 18 Sept 2018

  • A meta-analysis identified that physician burnout is associated with twofold increased odds of unsafe care, unprofessional behaviours and lower patient satisfaction. This results in inefficient and costly care delivery for healthcare systems. JAMA Internal Medicine. 1 Oct 2018

  • In a pilot randomised clinical trial, integration of formal stress resilience and mindfulness training into surgical residency appeared feasible and acceptable to surgical interns. Mindfulness skills appeared to be lasting, as evidenced by continued independent practice over 12 months of follow-up. JAMA Surgery. 17 Oct 2018

  • Introduction

    Burnout, a term first...

    Remembering individual perspectives and needs in differentiated HIV care strategies

    In 2018, the HIV pandemic response has transitioned from an emergency approach to one that will be implemented over several decades and likely for our lifetimes. An estimated 37 million people now live with HIV, of whom 21.5 million people are on antiretroviral therapy (ART) including an estimated 17.5 million people who have achieved viral suppression.1 Thus, an estimated 20 million people either require ART or improved regimens and/or adherence interventions.1 Though HIV diagnostic and prevention strategies are improving, stigma, implementation challenges and late diagnoses resulted in 1.8 million people acquiring HIV, indicating that HIV incidence has only been slowly decreasing over the last several years.1 Thus, the numbers of people requiring HIV treatment are likely to continue to increase over the coming years.

    Effective support for the lifelong treatment needs of an increasing number of people living with HIV despite decreasing levels...

    Workflow disruptions and surgical performance: past, present and future

    The earliest contributions of human factors engineering to surgery probably occurred just over a century ago. Already well-known for their pioneering scientific approaches to management, Frank and Lillian Gilbreth turned their expertise in time and motion studies and the psychology of management to optimising the operating room environment (and various aspects of hospital operations).1 Some readers may know this real husband and wife pair of early management gurus as the parents depicted in ‘Cheaper by the Dozen’. Written by two of their 12 children, it humorously depicted the application of the science of efficiency to the lives of this family of 14. The book, first published in 1946, generated a film (1950), a television series and an updated film in 2003. None of the antics in these various productions convey the real-life contributions of either of the Gilbreths. For instance, the modern approach to laying out surgical...

    Frailty and mortality: 'Same-same but Different

    Prediction models hold tremendous promise as a way to improve patient outcomes and healthcare quality more generally by efficiently targeting interventions to those patients most likely to benefit. Most aspects of healthcare (ie, tests, medications and procedures) have associated risks and burdens. Accurate prediction of patients at higher risk for adverse outcomes from their underlying conditions allows for better targeting of interventions, so that only patients for whom the benefits of the intervention outweigh the risks of treatment are provided the intervention. Conversely, accurate prediction of patients at low risk for adverse outcomes (for whom the risks of the intervention outweigh the benefits) can help those patients avoid unnecessary and potentially harmful interventions. Thus, accurate prediction models play a pivotal role in the vision of personalised medicine, where all clinical decisions are tailored to each individual’s unique risk profile.1

    In this issue of BMJ Quality & Safety,...

    Patients perspectives on how to decrease the burden of treatment: a qualitative study of HIV care in sub-Saharan Africa

    Objective

    Patients living with HIV infection (PLWH) in sub-Saharan Africa face an important burden of treatment related to everything they do to take care of their health: doctor visits, tests, regular refills, travels, and so on. In this study, we involved PLWH in proposing ideas on how to decrease their burden of treatment and assessed to what extent these propositions could be implemented in care.

    Methods

    Adult PLWH recruited in three HIV care centres in Côte d’Ivoire participated in qualitative interviews starting with ‘What do you believe are the most important things to change in your care to improve your burden of treatment?’ Two independent investigators conducted a thematic analysis to identify and classify patients' propositions to decrease their burden of treatment. A group of experts involving patients, health professionals, hospital leaders and policymakers evaluated each patient proposition to assess its feasibility.

    Results

    Between February and April 2017, 326 participants shared 748 ideas to decrease their burden of treatment. These ideas were grouped into 59 unique patient propositions to improve their personal care and the organisation of their hospital or clinic and/or the health system. Experts considered that 27 (46%), 19 (32%) and 13 (22%) of patients' propositions were easy, moderate and difficult, respectively, to implement. A total of 118 (36%) participants offered at least one proposition considered easily implementable by our experts.

    Conclusion

    Asking PLWH in sub-Saharan Africa about how their care could be improved led to identifying meaningful propositions. According to experts, half of the ideas identified could be implemented easily at low cost for minimally disruptive HIV care.

    Minor flow disruptions, traffic-related factors and their effect on major flow disruptions in the operating room

    Background

    Studies in operating rooms (OR) show that minor disruptions tend to group together to result in serious adverse events such as surgical errors. Understanding the characteristics of these minor flow disruptions (FD) that impact major events is important in order to proactively design safer systems

    Objective

    The purpose of this study is to use a systems approach to investigate the aetiology of minor and major FDs in ORs in terms of the people involved, tasks performed and OR traffic, as well as the location of FDs and other environmental characteristics of the OR that may contribute to these disruptions.

    Methods

    Using direct observation and classification of FDs via video recordings of 28 surgical procedures, this study modelled the impact of a range of system factors—location of minor FDs, roles of staff members involved in FDs, type of staff activities as well as OR traffic-related factors—on major FDs in the OR.

    Results

    The rate of major FDs increases as the rate of minor FDs increases, especially in the context of equipment-related FDs, and specific physical locations in the OR. Circulating nurse-related minor FDs and minor FDs that took place in the transitional zone 2, near the foot of the surgical table, were also related to an increase in the rate of major FDs. This study also found that more major and minor FDs took place in the anaesthesia zone compared with all other OR zones. Layout-related disruptions comprised more than half of all observed FDs.

    Conclusion

    Room design and layout issues may create barriers to task performance, potentially contributing to the escalation of FDs in the OR.

    External validation of the Hospital Frailty Risk Score and comparison with the Hospital-patient One-year Mortality Risk Score to predict outcomes in elderly hospitalised patients: a retrospective cohort study

    Objective

    Frailty is an important prognostic factor in hospitalised patients but typically requires face-to-face assessment by trained observers to detect. Thus, frail patients are not readily apparent from a systems perspective for those interested in implementing quality improvement measures to optimise their outcomes. This study was designed to externally validate and compare two recently described tools using administrative data as potential markers for frailty: the Hospital Frailty Risk Score (HFRS) and the Hospital-patient One-year Mortality Risk (HOMR) Score.

    Design

    Retrospective cohort study.

    Setting

    Ontario, Canada.

    Participants

    All patients over 75 with at least one urgent non-psychiatric hospitalisation between 2004 and 2010.

    Main outcome measures

    Prolonged hospital length of stay (>10 days), 30-day mortality after admission and 30-day postdischarge rates of urgent readmission or emergency department (ED) visits.

    Results

    In 452 785 patients (25.9% with intermediate or high-risk HFRS), increased HFRS was associated with higher Charlson scores, older age and decreased likelihood of baseline independence. Patients with high or intermediate HFRS had significantly increased risks of prolonged hospitalisation (70.0% (OR 8.64, 95% CI 8.30 to 8.99) or 49.7% (OR 3.66, 95% CI 3.60 to 3.71) vs 21.3% in low-risk HFRS group) and 30-day mortality (15.5% (OR 1.27, 95% CI 1.20 to 1.33) or 16.8% (OR 1.39, 95% CI 1.36 to 1.41) vs 12.7% in low-risk), but decreased risks of 30-day readmission (10.0% (OR 0.74, 95% CI 0.69 to 0.79) and 11.2% (OR 0.84, 95% CI 0.82 to 0.86) vs 13.1%) or ED visit (7.3% (OR 0.41, 95% CI 0.38 to 0.45) and 11.1% (OR 0.66, 95% CI 0.38 to 0.45) vs 16.0%). Although only loosely associated (Pearson correlation coefficient 0.265, p<0.0001), both the HFRS and HOMR Score were independently associated with each outcome—HFRS was more strongly associated with prolonged length of stay (C-statistic 0.71) and HOMR Score was more strongly associated with 30-day mortality (C-statistic 0.71). Both poorly predicted 30-day readmissions (C-statistics 0.52 for HFRS and 0.54 for HOMR Score).

    Conclusions

    The HFRS best identified hospitalised older patients at higher risk of prolonged length of stay and the HOMR score better predicted 30-day mortality. However, neither score was suitable for predicting risk of readmission or ED visit in the 30 days after discharge. Thus, a single score is inadequate to prognosticate for all outcomes associated with frailty.

    Recognising the importance of informal communication events in improving collaborative care

    Background

    While the concept of collaboration is highly touted in the literature, most descriptions of effective collaboration highlight formal collaborative events; largely ignored are the informal collaborative events and none focusing on the frequent, ‘seemingly’ by chance communication events that arise and their role in supporting patient safety and quality care.

    Objective

    To identify the types of informal communication events that exist in the inpatient setting and better understand the barriers contributing to their necessity.

    Methods

    We undertook a constructivist grounded theory study in an inpatient internal medicine teaching unit in Ontario, Canada. Interview and observational data were collected across two phases; in total, 56 participants were consented for the study. Data collection and analysis occurred iteratively; themes were identified using constant comparison methods.

    Results

    Several types of informal communication events were identified and appeared valuable in three ways: (1) providing a better sense of a patient’s baseline function in comparison to their current function; (2) gaining a more holistic understanding of the patient’s needs; and (3) generating better insight into a patient’s wishes and goals of care. Participants identified a number of organisational and communication challenges leading to the need for informal communication events. These included: scheduling, competing demands and the spatial and temporal organisation of the ward. As a result, nursing staff, allied health professionals and caregivers had to develop strategies for interacting with the physician team.

    Conclusion

    We highlight the importance of informal communication in supporting patient care and the gaps in the system contributing to their necessity. Changes at the system level are needed to ensure we are not leaving important collaborative opportunities to chance alone.

    Pages