Course Content
Site: | SLMTA |
Course: | Statistical Quality Control and Method Validation/Verification (QCMV2) e-Learning |
Book: | Course Content |
Printed by: | Guest user |
Date: | Thursday, 5 December 2024, 6:53 PM |
Table of contents
- 0 - Workshop Introduction
- 1 - Let's Examine the Basics
- 2 - Gaussian is the key
- 3 - Match it up
- 4 - The Front line worker
- 5 - It Begins with the Right Chart
- 6 - Paralell testing
- 7 - Total Error
- 8 - Putting TEa into Quality
- 9 - How Far can Your Mean Shift
- 10 - How to Select Control Rules
- 11 - How Proficient Are We
- 12 - Understanding Interlab Comparison
- 13 - Introduction to Method Validation and Verification
- 13.1 Introduction Part 1
- 13.2 Introduction Part 2
- 13.3 ISO Requirements
- 13.4 Intro Summary
- 13.5 Performance Characteristics 1
- 13.6 Performance Characteristics 2
- 13.7 Performance Characteristics 3
- 13.8 Performance Characteristics 4
- 13.9 Performance Characteristics 5
- 13.10 Validation and Verification Summary
- 13.11 Introduce the Activity - Decision Tree
- 13.12 MV Overview Part 1
- 13.13 MV Overview Part 2
- 13.14 MV Overview Part 3
- 13.15 Precision Experiment 1
- 13.16 Precision Experiment 2
- 13.17 Intro the Activity - Precision
- 13.18 Linearity Experiment 1
- 13.19 Linearity Experiment 2
- 13.20 Linearity Experiment 3
- 13.21 Linearity Experiment 4
- 13.22 Linearity Experiment 5
- 13.23 Linearity Experiment 6
- 13.24 Linearity Experiment 7
- 13.25 Intro the Activity-Linearity
- 13.26 Accuracy Experiment 1
- 13.27 Accuracy Experiment 2
- 13.28 Accuracy Experiment 3
- 13.29 Accuracy Experiment 4
- 13.30 Accuracy Experiment 5
- 13.31 Accuracy Experiment 6
- 13.32 Accuracy Experiment 7
- 13.33 Accuracy Experiment 8
- 13.34 Accuracy Experiment 9
- 13.35 Accuracy Experiment 10
- 13.36 Accuracy Experiment 11
- 13.37 Introduce the Activity - Accuracy
- 13.38 Reference Interval Experiment 1
- 13.39 Reference Interval Experiment 2
- 13.40 Reference Interval Experiment 3
- 13.41 Reference Interval Experiment 4
- 13.42 Introduce the Activity - Ref Interval
- 14 - Measurement of Uncertainty
- 14.1 Introduction
- 14.2 ISO Requirements
- 14.3 TE vs MU
- 14.4 MU Part 1
- 14.5 MU Part 2
- 14.6 MU Part 3
- 14.7 Introduce the Activity - MU Calculation
- 14.8 Traceability 1
- 14.9 Traceability 2
- 14.10 Commutable
- 14.11 Apply MU
- 14.12 Diagnostic Uncertainty
- 14.13 Intro the Activity - Reliable Repeat
- 14.14 Intro the Activity - Using the Rule of 3
- 15 - Improvement Project Master Class
0 - Workshop Introduction
Overview
Purpose
Participants are provided with an overview of how this workshop integrates quality control (QC) into their laboratory to support their accreditation efforts. Additionally, workshop expectations are presented to participants.
Key Message
- The workshop targets primarily quantitative QC and method evaluation.
- Linkages between QC workshop training and ISO 15189 will be made throughout the workshop.
- Where are we, where do we expect to be, and where do we want to be are three questions that will be used throughout the workshop to assist with data analysis and evaluation.
Self Assessment
Can you:
- Understand how the QC workshop supports the SLMTA process?
- Understand the workshop’s scope, targeting quantitative QC?
- Understand expectations during and after the workshop?
Resources
- Handout: LQMS Content Sheet
- Worksheet: Measures of Central Tendency NEW
- Basic lessons in Lab QC (Booklet)
- Final Examination Answer Template
- Job Aid 1: Navigating the Westgard Internet Site NEW
1 - Let's Examine the Basics
Overview
Purpose
In this activity, the pre-requisite reading’s Final Examination on basic QC and Measures of Central Tendency worksheet will be reviewed with participants. Tools to calculate mean, standard deviation (SD), and coefficient of variation (CV) will be introduced and used throughout the workshop.
Key Message
- The mean, SD, and %CV are foundational statistics used with QC
- Given a data set, the mean, SD, and %CV can be calculated.
- Calculators, spreadsheets, and online resources are available tools sites can use with their QC data.
Self Assessment
Can you:
- Calculate the mean, SD and %CV from a given data set?
- Utilize the tools introduced to obtain a mean, SD, and %CV from their site’s data set?
- Calculate the measures of central tendency?
Resources
2 - Gaussian is the key
Overview
Purpose
Stable analytical systems will produce the same Gaussian distribution of data. When a system undergoes a change, an unexpected data point will be produced. In this activity, participants will demonstrate the foundational principles of statistical QC (SQC).
Key Message
- The accuracy and precision of an analytical system describes WHERE WE ARE.
- The basis of SQC is knowing WHERE WE ARE and WHERE WE EXPECT TO BE.
- If WHERE WE ARE with our current observed value is not WHERE WE EXPECT TO BE, then a change has occurred.
Self Assessment
Can you:
- List the performance specifications that describe WHERE WE ARE with an analytical system?
- Define SQC and how it applies to monitoring their analytical system?
- Define the criteria required for SQC to monitor change?
Resources
2.1 Gaussian is key
2.2 Measures of Central Tendency
2.3 Gaussian distribution
2.4 Gaussian demonstration
2.5 Levey Jenings Chart
2.6 Data populations
2.7 SQC recap
3 - Match it up
Overview
Purpose
QC rules alert laboratorians to a change in the analytical system (measurement procedure). In this activity, participants match charts to the correct QC rule violation and identify the type of error.
Key Message
- Results of QC samples are analyzed to assess and alert us to changes in accuracy and precision (method performance).
- The nomenclature for rule violations includes the number of data points involved and the applied control limits usually expressed as the mean ± a multiple of the SD.
- Different rules indicate if the change is random or systematic. This information is used to assist with troubleshooting an out-of-control run.
Self Assessment
Can you:
- Analyze L-J charts and Gaussian distributions and identify the rule violation?
- Determine if the rule indicates a systematic or random error?
- Understand how SE and RE assists with troubleshooting?
Resources
- Handout 1_Bell Curve with Distributions (304).pptx
- Handout 2_L-J Chart with Distributions (305).pptx
- Worksheet - Match it up 3 Terms and concepts.docx
- Tool 1_Rule Violations 3.3 New.pptx
- Worksheet _Rule Violation Answer Sheet (306) NEW.docx
3.1 Match it up Introduction
3.2 Common QC terms
3.3 Homework Part 1 Match it up
3.4.1 Homework Part II Introduction Match it up
3.4.2 Homework Part II Match it up
4 - The Front line worker
Overview
Purpose
Front-line workers assess whether or not an analytical run is acceptable. QC protocols developed by laboratory management must standardize the decision criteria used by the workers. In this activity, participants are provided with QC charts and a Westgard Multirule algorithm and must apply the decision-making process to determine acceptability of the analytical run.
Key Message
- Multirules provide a high level of error detection capability while at the same time keeping false rejection rates low.
- Laboratory management must provide clear instructions to the front-line workers regarding QC decision criteria.
- The 1:2s warning rule is a scanning tool that can save time and effort in manual QC applications.
Self Assessment
Can you:
- Determine whether an analytical run is acceptable or not based upon a given QC protocol?
- Recognize how the 1:2s warning rule can be used as a scanning tool?
- Understand when a multirule should be applied to a measurement procedure?
Resources
- Handout_Westgard Rules (403).pdf
- Job Aid_Defining a QC Protocol (407).docx
- Worksheet 1_Scenario 1 (404).xls
- (NEW) Worksheet 2_Scenario 2 (405) 2024.xls
- Worksheet 3_Scenario 3 (406).xls
4.1 Introduction to The Front line worker
4.2 Homework The Front line worker
5 - It Begins with the Right Chart
Overview
Purpose
Using the QC tool, L-J Chart, one can visually inspect a quantitative method’s current accuracy and precision to quickly assess if a change is occurring within the system. In this activity, participants learn how to correctly create this tool, how to visually assess it, and how to avoid pitfalls commonly encountered with its set-up.
Key Message
- One of the most widely used tools with QC is the L-J chart. This tool allows one to visually assess for a change in a method’s current performance.
- When L-J charts are regularly examined, a small change may be recognized long before a QC rule violation occurs.
- If the assigned values used on the chart do not reflect the observed values of the current performance, then the QC rules are unable to function properly to detect errors. The data will no longer appear normally distributed.
Self Assessment
Can you:
- Create a properly prepared L-J chart?
- Visually assess a chart to see if there has been a change in the method’s performance?
- Recognize common pitfalls when the assigned values do not equal the observed values of a stable system?
Resources
- Job Aid 1_Possible Outcomes (510).pptx
- Job Aid 2_Creating a L-J Chart (511).pptx
- Job Aid 3_Assigned vs.Observed (512).pptx
- Worksheet 1_L-J Chart 1 (504).docx
- (NEW) Worksheet 2_L-J Chart 2 (505) 2024.docx
- Worksheet 3_L-J Chart 3 (506).docx
- Worksheet 4_L-J Chart 4 (507).docx
- Worksheet 5_L-J Chart 5 (508).docx
- (NEW) Worksheet 6_L-J Chart 6 (509) 2024.docx
5.1 Introdcution It Begins with the Right Chart
5.2 True and False Accepts and Rejects
5.3 LJ Chart set up
5.4 Homework Part I Assigned=Observed
5.5 Homework Introduction to Part II Assigned ≠Observed
5.6 Homework Part II Assigned ≠ Observed
6 - Paralell testing
Overview
Purpose
Only when the system is known to be stable can data be collected to calculate the observed mean and SD of a new lot number of QC material. % CV is introduced and used in the SD computation of short shelf-life control materials.
Key Message
- New lot numbers of control material should be tested in parallel with the currently in-use control material.
- Parallel testing is only performed when the instrument is operationally stable.
- Package inserts serve as a guideline only and should not be used when creating the L-J chart for the new control. %CV tells us WHERE WE ARE in terms of precision.
Self Assessment
Can you:
- Understand the importance of parallel testing?
- Identify the steps to perform parallel testing at their site?
- Calculate the %CV?
Resources
- Homework Part I Worksheet_Applying %CV (605).docx
- (NEW) Homework Part II Worksheet the Great Equaliser 2024.docx
- Job Aid 1_Selecting Appropriate QC Material (606).docx
- Job Aid 2_Parallel Testing (607).docx
6.1 Paralell testing QC material
6.2 Homework CV%
7 - Total Error
Overview
Purpose
Total error (TE) combines bias and imprecision to quantify the largest variation from the true (target) value. In this activity, participants are introduced to bias and true value. Using the information supplied by 3 key numbers, participants calculate the TE in units and percent. Additionally, participants explore the impact that bias and imprecision can have on TE and relate site-specific processes that affect TE to checklist items.
Key Message
- 3 key numbers are required to calculate TE – mean, SD, and true value.
- The target value is our best estimate using sources that each has its own limitation.
- TE is the total variation of our value from the true value.
Self Assessment
Can you:
- Calculate bias and absolute bias?
- Calculate TE in units and percent?
- Identify ways to reduce bias and imprecision at their site?
Resources
- (New) Worksheet - It Begins with the Right Chart & TE Terms and Concepts.docx
- Job Aid_How to Calculate TE (705).pptx
- Worksheet 1_Calculate TE (703).docx
- Worksheet 2_Impact of Bias and Imprecision on TE (704) NEW.docx
7.1 Introduction to Total Error
7.2 Bias equation TE
7.3 TE Calculation Part I
7.4 TE Calculation Part II
7.5 Homework Calculating Total Error
8 - Putting TEa into Quality
Overview
Purpose
To control quality, one must first determine what quality is needed for a test. In this activity, participants will explore what information the fourth key number must provide to define the quality required. Additionally, participants will use a laboratory scenario to assess if the method remains acceptable for the intended clinical use of the test result.
Key Message
- TEA defines the quality specifications specific to the analyte being tested to make the analytical result clinically meaningful.
- QC statistics can effectively reflect method performance if acceptable performance limits are defined and applied.
- For clinically meaningful results, the TE should be less than the TEA for that analyte.
Self Assessment
Can you:
- Identify the four key numbers needed for quality?
- Calculate error limits if given the Target Value and TEA?
- Compare TE to TEA to determine if the method still performs within acceptable limits?
Resources
- Job Aid 1_ Using TEA (813).pptx
- Job Aid 2_Sample TEA calculations (814).pptx
- Worksheet 1_Intended Use (805).docx
- Worksheet 2A_Intended Use (806).docx
- Worksheet 2B_Intended Use (807).docx
- Worksheet 3A_Intended Use (808).docx
- Worksheet 3B_Intended Use (809).docx
- Worksheet 4_Intended Use (810).docx
- Worksheet 5_Calculating TEA Error Limits (811) NEW.docx
- Worksheet 6_Reagent Lot Number Change (812) NEW.docx
8.1 Homework Intended use Putting TEa into Quality
8.2 Using Total Allowable Error
8.3 Calcuating TEa limits
8.4 Homework TEa Calculations and Reagent lot number changes
9 - How Far can Your Mean Shift
Overview
Purpose
Benchmarking measurement procedures using TEA and the current method’s performance allows a laboratory to identify which methods are meeting quality specifications and which are not. In this activity, Sigma-metric and Critical Systematic Error (ΔSEc) are introduced. Participants will review a site’s monthly summary report and perform an investigational analysis using instrument records and QC charts.
Key Message
- SEc indicates how far the mean can shift before quality performance requirements are exceeded.
- Benchmarking methods using a Sigma scale allows a laboratory to use their resources wisely.
- Good recordkeeping is essential in resolving QC problems.
Self Assessment
Can you:
- Describe how SEc and Sigma indicate method performance relative to quality performance goals?
- Calculate SEc and Sigma?
- Use information gleaned from records to troubleshoot a QC issue?
Resource
- Handout Monthly Review (905).xlsx
- Worksheet TEa 8 and How far can your mean shift 9 terms and concepts.docx
- Job Aid 1_Frequency of Chart Review(909).pdf
- Job Aid 2_Daily QC Investigation Template (910).docx
- Job Aid 3_QC NCE Examinations (911v2).pdf
- Tool 2_Investigation Records (903).docx
- Worksheet 1_Calculating SEc and Sigma (906).docx
- Worksheet 2 QC Investigation (907).docx
- Worksheet 3 Case Scenario (908).doc
9.1 Introduction to How Far can Your Mean Shift
9.2 Introduction to SEc
9.3 Calculating SEc
9.4 Introduction to Sigma metrics
9.5 Homework Calculating SEc and Sigma
9.6 Homework Review and investigate
10 - How to Select Control Rules
Overview
Purpose
Laboratories must select appropriate QC rules based on the quality required for a test and the method’s observed accuracy and precision. In this activity, participants are introduced to the Sigma-metrics QC Selection Tool, available at Westgard.com. Using a case scenario, participants will select appropriate QC rules and design a QC strategy for a workstation.
Key Message
- The measurement procedure’s quality requirement and the accuracy and precision must be considered in the selection of appropriate QC rules.
- The control rule selected should alert the laboratorian to a significant change before wrong results are reported.
- Selected control rules should maximize the error detection (Ped ≥ 90%) and minimize the false rejection (Pfr ≤ 5%) while using the lowest number of control measurements possible.
Self Assessment
Can you:
- Select appropriate QC rules using the Sigma-metrics QC Selection Tool from the Westgard internet site?
- Select a TEA% using the Westgard internet site?
- Design an effective QC strategy for a workstation?
Resources
- Handout 1_QC Rule Selection (1003).pdf
- Handout 2_ Monthly Review Log (1004).xlsx
- Job Aid 1_Frequency of Chart Review(909).pdf
- Job Aid 2_Steps for Developing a QC Strategy (1009).docx
- Job Aid_Defining a QC Protocol (407).docx
- Worksheet 1_Predicting Ped Capabilities (1006).docx
- Worksheet 2_Candidate QC Rules (1007v2) NEW.docx
10.1 Introduction to How to Select Control Rules
10.2 QC selection tool introduction Part I
10.3 QC selection tool introduction Part II
10.4 Homework Predicting Probabilities
10.5 Homework Workstation activity
11 - How Proficient Are We
Overview
Purpose
External Quality Assessment Schemes (EQAS) enable the laboratory to assess on-going accuracy by comparing its method’s performance to external sources. In this activity, participants are introduced to standard deviation index (SDI) and z-score and will review a proficiency test (PT) report.
Key Message
- Proficiency testing (PT) programs provide a way to verify long-term accuracy of a method.
- SDI is a quick indicator of how well our mean compares to the method peer mean.
- All PT failures must be investigated and documented by the laboratory.
Self Assessment
Can you:
- Recognize the common elements found in a PT report and their significance?
- Calculate SDI and use that information to assess the accuracy of the method?
- Recognize that PT results, both acceptable and unacceptable, should be evaluated?
Resources
- Handout_PT Examples (1105).pdf
- Homework- PT Reading Assignment Quiz.doc
- Job Aid 1_General Guidelines for Proficiency Testing (1107).pdf
- Job Aid 2_ Interpreting PT Reports (1108).pdf
- Job Aid 3_PT Failure Checklist (1109).pdf
- Tool 2_ Comparison of Report Formats (1103).docx
- Worksheet_PT Calculations (1106).docx
11.1 Why EQA Part 1
11.2 Why EQA Part 2
11.3 PT Programs Part 1
11.4 PT Programs Part 2
11.5 PT Programs Part 3
11.6 PT Programs Part 4
11.7 PT Programs Part 5
11.8 Introduce the Activity
12 - Understanding Interlab Comparison
Overview
Purpose
Laboratories using the same measurement procedure and analyzing the same control material will generate similar means and SDs. This information is a valuable source to determine the target value of the QC material for a laboratory and compare their method’s performance to their peers.
In this activity, participants are introduced to coefficient of variation index (CVI). Based on the information supplied from the inter-laboratory comparison report, participants will investigate a QC problem.
Key Message
- Inter-laboratory peer comparisons can help identify method and PT problems.
- The peer mean is an excellent source of the true (target) value for each control.
- Coefficient of Variation Index is a peer-based metric of imprecision.
Self Assessment
Can you:
- Calculate the SDI and CVI?
- Verify the 4 key numbers used to monitor and evaluate performance?
- Perform an investigational analysis regarding a summary statistic flag?
Resources
- Handout 2_ Alk Phos Report (1208).pdf
- Handout 3_ Calcium Report (1209).pdf
- Homework- Interlab Reading Assignment Quiz.doc
- Job Aid 1_ Comparison of EQA Programs (1214).pdf
- Job Aid 2_ Interpreting Comparison Reports (1215).pdf
- Worksheet 1A_ Calculating SDI and CVI.docx
12.1 Introduce
12.2 Comparison
12.3 QC Materials
12.4 How It Works
12.5 CVI Part 1
12.6 CVI Part 2
12.7 CVI Part 3
12.8 Introduce the Activity
13 - Introduction to Method Validation and Verification
Overview
Purpose
Before a method can be placed into routine service, it must be evaluated to ensure that the measurement procedure meets defined criteria, such as sensitivity, specificity, precision, accuracy, and linearity. In this activity, participants will perform method verification experiments and assess data to determine if the method is acceptable or not.
Key Message
- Even though manufacturers test their methods extensively, there are many factors in an individual laboratory that can affect the actual precision and accuracy of the method.
- Method evaluation is performed to reveal the amount of error present in the new method.
- Once the performance of the method has been judged to be acceptable from method evaluation studies, then statistical QC procedures need to be selected that can detect medically important errors.
Self Assessment
Can you:
- List the steps required to introduce a new method?
- Collect and analyze the experimental data needed to quantify the error present?
- Judge if the method meets pre-defined quality requirements?
- Verify a reference interval?
Resources
- General Resources
- Linearity Experiment
- Precision Experiment
- Comparison of Method Experiment
- Reference Interval Experiment
13.1 Introduction Part 1
13.2 Introduction Part 2
13.3 ISO Requirements
13.4 Intro Summary
13.5 Performance Characteristics 1
13.6 Performance Characteristics 2
13.7 Performance Characteristics 3
13.8 Performance Characteristics 4
13.9 Performance Characteristics 5
13.10 Validation and Verification Summary
13.11 Introduce the Activity - Decision Tree
13.12 MV Overview Part 1
13.13 MV Overview Part 2
13.14 MV Overview Part 3
13.15 Precision Experiment 1
13.16 Precision Experiment 2
13.17 Intro the Activity - Precision
13.18 Linearity Experiment 1
13.19 Linearity Experiment 2
13.20 Linearity Experiment 3
13.21 Linearity Experiment 4
13.22 Linearity Experiment 5
13.23 Linearity Experiment 6
13.24 Linearity Experiment 7
13.25 Intro the Activity-Linearity
13.26 Accuracy Experiment 1
13.27 Accuracy Experiment 2
13.28 Accuracy Experiment 3
13.29 Accuracy Experiment 4
13.30 Accuracy Experiment 5
13.31 Accuracy Experiment 6
13.32 Accuracy Experiment 7
13.33 Accuracy Experiment 8
13.34 Accuracy Experiment 9
13.35 Accuracy Experiment 10
13.36 Accuracy Experiment 11
13.37 Introduce the Activity - Accuracy
13.38 Reference Interval Experiment 1
13.39 Reference Interval Experiment 2
13.40 Reference Interval Experiment 3
13.41 Reference Interval Experiment 4
13.42 Introduce the Activity - Ref Interval
14 - Measurement of Uncertainty
Overview
Purpose
Every result has some degree of uncertainty that comes from the calibration of the method and the analysis itself. In this activity, participants learn how to determine measurement uncertainty using a top-down approach and how to apply measurement uncertainty in an advisory capacity for their clients.
Key Message
- The uncertainty gives the limits of the range in which the true value of the measurand is estimated to be at a given probability.
- The long-term %CV or SD data gained through the laboratory’s IQC is sufficient for satisfying the minimal criteria for MU determination.
- The MU is helpful in differentiating whether a change in a serial test result is due to analytical variation or a true physiological change.
Self Assessment
Can you:
- Recognize the need for hospital management support and analytical staff adherence when planning a QC program?
- Identify the steps needed during the planning phase to successfully implement a QC program?
- Further refine the process plan to identify what happens, who’s responsible, and by when?
Resources
- Basic of Estimating MU.pdf
- Hitchhikers Guide to MU.pdf
- Homework- MU Reading Assignment Quiz.doc
- Job Aid 1_Uncertainty Hierarchy (1505v2).pdf
- Job Aid 2_Traceability (1506v2).pdf
- Job Aid 3_Diagnostic Uncertainty (1507v2).pdf
- Worksheet Reliable Repeat.ppt
- Worksheet Using Rule of 3.ppt
- Worksheet_MU Calculations (v3).pptx
14.1 Introduction
14.2 ISO Requirements
14.3 TE vs MU
14.4 MU Part 1
14.5 MU Part 2
14.6 MU Part 3
14.7 Introduce the Activity - MU Calculation
14.8 Traceability 1
14.9 Traceability 2
14.10 Commutable
14.11 Apply MU
14.12 Diagnostic Uncertainty
14.13 Intro the Activity - Reliable Repeat
14.14 Intro the Activity - Using the Rule of 3
15 - Improvement Project Master Class
15 - Improvement Project Master Class
15.1 Improvement Project Master Class
- Overview
Purpose
The knowledge gained from a workshop becomes effective when the improvement is applied at the site for better patient care. In this activity, participants will map the process needed to perform an improvement project (IP) assignment.
Key Message
- The planning phase is essential in any improvement project.
- The support of hospital management and analytical staff is essential for implementing an effective QC program.
- The QC IP plan involves many sections of the accreditation checklist.
Self Assessment
Can you:
- Recognize the need for hospital management support and analytical staff adherence when planning a QC program?
- Identify the steps needed during the planning phase to successfully implement a QC program?
- Further refine the process plan to identify what happens, who’s responsible, and by when?