SCIE90011 Validation Studies Flashcards

SCIE90011: From Lab to Life - Validation Studies

Intended Learning Outcomes

By the end of this lecture, students should be able to:

• Distinguish verification from validation.

• Explain the role of validation studies in demonstrating conformance to User Requirement Specifications (URS) and Critical Quality Attributes (CQAs).

• Identify the main types of validation studies.

• Outline key design elements of a validation study: objectives, endpoints, populations/materials, controls/comparators, sample size considerations, bias, and robustness.

• Understand how risk, regulation, and uncertainty influence validation strategy.

Understanding Validation and Verification

Validation and testing are critical stages to ensure the product meets all requirements and performs as expected.

General Process

• Process: The product undergoes rigorous testing to validate functionality, quality, and compliance with regulatory standards. This stage includes user testing, quality assurance, and iterative improvements based on feedback.

• Outcome: A validated product ready for market launch, with any identified issues or limitations addressed.

The Core Distinction

• Verification: "Did we build the thing right?"

  • Checking that the design outputs meet design inputs and specifications.

• Validation: "Did we build the right thing?"

  • Demonstrating that the final product or process meets user needs and intended use in a real or simulated-real context.

Types of Validation Studies

Analytical / Assay Validation

Relevant for diagnostics, bioanalytics, and in-process controls.

• Purpose: To validate measurement methods used to generate data for product performance, release, or process control.

• Typical Parameters:

  • Accuracy and precision (including repeatability and reproducibility).

  • Sensitivity, specifically Limit of Detection (LoD) and Limit of Quantitation (LoQ).

  • Linearity and range.

  • Specificity and selectivity.

  • Robustness (performance under small variations in conditions).

Process Validation

Relevant for manufacturing and bioprocessing.

• Purpose: To demonstrate that a defined process, when operated within specified ranges, consistently produces a product meeting CQAs.

• Typical Elements:

  • Process Performance Qualification (PPQ) runs: For example, successfully completing $3$ consecutive commercial-scale batches.

  • Critical Process Parameters (CPPs): Demonstration of control over these parameters.

  • CQA Evidence: Proof that CQAs are met across all PPQ runs.

• Regulatory Context: Good Manufacturing Practice (GMP) expects validated processes to be shown as capable and under control over multiple runs, rather than a single occurrence.

Product / Performance Validation

This answers the question: "Does this product deliver the claimed clinical/functional performance in real or realistically simulated use?"

• Diagnostics: Clinical performance studies against a reference standard including sensitivity, specificity, predictive values, and likelihood ratios. Field evaluations take place in intended settings like Emergency Departments (ED), primary care, or low-resource settings.

• Therapeutic Products: Clinical trials (Phase II and Phase III) to demonstrate safety and efficacy.

• Performance Validation of Devices: Bench testing under clinically relevant conditions and field trials in the intended user environment.

Usability / Human Factors Validation

Especially important for devices and complex workflows.

• Purpose: To show that intended users, in the intended environment, can use the product safely and effectively without unacceptable use errors.

• Elements:

  • Involvement of representative users.

  • Realistic tasks and use scenarios.

  • Observation of errors, near-misses, and workarounds.

  • Analysis and remediation of use-related risks.

• Connections: Links back to URS on usability (e.g., training time, steps, acceptable error rates) and risk management (e.g., ISO 14971 for medtech).

Core Elements of Designing a Validation Study

Step 1: Define the Validation Question

Determine what must be shown to claim that URS and intended use are met. Examples include:

• Does a diagnostic test identify condition X with at least $Y\%$ sensitivity and $Z\%$ specificity in the target population?

• Does the upstream process at commercial scale produce batches where $\ge 95\%$ meet all CQAs?

• Can nurses in ED triage correctly perform the test with no more than $W\%$ critical use errors after standard training?

Step 2: Identify Endpoints and Success Criteria

• Endpoints: Observable, measurable outcomes used to evaluate performance.

  • Diagnostic examples: Primary endpoints (sensitivity, specificity vs. gold standard); Secondary endpoints (time-to-result, invalid test rate).

  • Process examples: Proportion of batches within all CQA limits.

  • Usability examples: Number and type of critical use errors, task completion time, need for assistance.

• Success Criteria: Pre-defined thresholds drawn from URS, guidelines, or risk analysis. Example: "Lower $95\%$ confidence bound for sensitivity $\ge 90\%$".

Step 3: Define Context, Population, and Materials

• Clinical/Product Validation: Define the target population (patients, healthy volunteers, clinicians, or technicians) with specific inclusion/exclusion criteria. Represent settings such as tertiary centers vs. regional hospitals.

• Process Validation: Include the range of anticipated process conditions, such as raw material lots and environmental conditions.

• Analytical Validation: Use varied matrices (serum, plasma, whole blood) and identify interference sources. Account for operator skill levels (typical lab tech vs. expert).

• Rule: Avoid "perfect" conditions; represent realistic variability.

Step 4: Design Structure (Controls, Comparators, and Bias Control)

• Controls/Comparators: Utilize gold standard tests, reference methods, current standard of care, or negative/positive controls in assays.

• Bias Control: Implement blinding where feasible (e.g., clinicians blinded to index test results). Choose between prospective or retrospective designs and use randomization where applicable. Avoid "cherry-picking" easy samples or ideal settings.

Step 5: Sample Size and Statistical Considerations

Sample size is dependent on:

• Expected effect or performance level (e.g., sensitivity).

• Desired confidence in estimates (e.g., width of confidence intervals).

• Acceptable risk of false conclusions (Type I and Type II errors).

• Scale: Small samples lead to wide uncertainty and unconvincing results. Larger samples provide more precise estimates and generalisability but require more resources.

Market Testing Strategies

Market testing evaluates a product's potential success by gathering feedback from a target audience before a full-scale launch.

Importance of Market Testing

• Identifies Potential Issues: Prevents costly mistakes by catching flaws before wider release.

• Gathers Customer Feedback: Refines products based on customer preferences and behaviors.

• Reduces Risk: High-control environments minimize the chance of failure.

• Optimizes Marketing Strategies: Identifies effective messages and channels.

• Informs Pricing Decisions: Gains insights into what customers are willing to pay.

• Enhances Product Development: Continuous feedback ensures the final product is well-received.

Specific Testing Strategies

• Surveys: Cost-effective method to reach large audiences quickly via email or social media (common in software).

• Focus Groups: Small group discussion guided by a moderator for in-depth insights into attitudes (common in consumer goods).

• Beta Testing: Releasing to a limited audience outside the company to find bugs and usability issues (tech industry standard).

• Test Marketing: Launching in a limited market to gauge performance and refine strategies.

• Product Demonstrations: Allowing customers to experience products firsthand (common in automotive and electronics).

• Observational Studies: Watching consumers interact with products in natural settings (retail and consumer goods).

• A/B Testing: Comparing two versions of a product or feature to optimize user experience and conversion rates.

Factors Affecting Market Testing Strategies

• Regulatory Requirements: Biotech/healthcare products must comply with standards, often requiring clinical trials.

• Product Type: Pharmaceuticals require trials, while diagnostics may benefit more from beta testing.

• Target Market: Demographics influence whether a survey or focus group is more effective.

• Budget and Resources: Large-scale surveys or trials require high investment; startups may use pilot studies.

• Regulatory Feedback: Early engagement helps ensure compliance and reduce delays.

• Technological Capabilities: Availability of digital platforms or advanced labs.

• Market Competition: High competition may necessitate more comprehensive testing to stand out.

• Time Constraints: Urgency may prioritize fast-to-market strategies like beta testing over longer methods.

Risk-Based and Regulatory Perspectives

Risk-Based Validation

Higher-risk products or processes require more rigorous validation. Risk factors include:

• Potential impact of failure on patient safety or product quality.

• Novelty of technology or indication.

• Complexity and number of failure modes.

Risk-based thinking determines the number of studies or runs, the stringency of success criteria, and which edge cases must be explicitly tested.

Regulatory Expectations

• Therapeutics: Evidence from phased clinical studies (safety, efficacy, dose-finding, confirmatory), validated analytical methods, and validated manufacturing processes.

• Diagnostics and Devices: Analytical validation (LoD, specificity), clinical performance in intended use settings, and usability/human factors validation.

• Bioprocesses: Process validation demonstrating consistency, ongoing process verification, and continuous monitoring.

Summary

• Validation demonstrates that a product or process meets user needs and intended use under realistic (not just ideal) conditions.

• Validation converts URS and designs into evidence that regulators, clinicians, manufacturers, and customers can trust.

• Without well-designed validation, technically brilliant ideas cannot safely move "from lab to life."