Developing an Experimental Plan

Topics Covered in Lecture

  • Overview of experimental design

  • Exploration of sources of experimental error

  • Discussion on accuracy and precision of measurements at various concentrations

  • Importance of replication in experimental research

Sources of Experimental Error

Importance of Minimizing Error

  • Experimental design should focus on minimizing noise and maximizing the chance of detecting true positives. The challenge is that often researchers focus too narrowly on certain aspects, akin to the phrase "can’t see the wood for the trees".

  • Variance can arise from treatments, random differences between samples/individuals, observations, analysis, or sample storage, which may obscure treatment effects.

Nature of Noise

  • Noise: This concept refers to the residual random spread of data in experimental results, which obscures the true effects of the treatment.

  • The nature of noise varies depending on the specific nature of the experiment being carried out.

Variability in Measurements

  • Greater variability complicates the detection of differences between means.

  • More measurements and replicates may be required, increasing workload yet decreasing the likelihood of detecting a genuine difference. The number of samples necessary for detecting a real difference increases as the means converge.

Types of Studies and Their Scope

Considerations in Study Design

  • Understand whether the study is experimental or observational.

  • Determine if it is field-based or lab-based.

  • Define the number of individuals, size of the site, or number of observations made.

  • Identify if the study focuses on cell/tissue/biochemical/molecular levels.

  • Generally, noise can be better controlled in laboratory settings compared to field studies.

Sources of Variance in Wheat Variety Trials

Identifying Types of Variance

A trial of wheat varieties might encompass the following quantitative measures, each associated with different sources of variance:

  1. Yield

  2. Protein and starch content

  3. Antioxidant profile

  4. Pesticide contamination

Analysis of Variance Sources
Yield

  • Biological/environmental variance arises as each variety reacts differently to environmental effects (denoted as G x E).

  • Sampling errors over the years and across different sites may manifest due to variations in harvesting techniques, leading to inefficiencies or losses.

  • Expected variance largely stems from G x E, assuming competent research methodology.

Starch and Protein Content

  • Variance attributes stem from biological/environmental factors (G x E).

  • Errors in sampling or harvesting primarily are insignificant for this parameter.

  • Measurement errors tend to be negligible when proper precautions are implemented as these involve straightforward assays.

Antioxidant Profile

  • Similar to yield, variance arises primarily from biological/environmental factors (G x E).

  • Sampling errors are typically not significant for this parameter.

  • Analytical errors can arise from operator differences, varying protocols, and equipment performance drift, but can be controlled with proper precautions. In this case, G x E is the main contributor to variance, though some may arise from analytical errors.

Pesticide Contamination

  • The sources of biological/environmental variance reflect the varying conditions under which plants are grown (G x E).

  • Errors due to sampling are less relevant to this measure.

  • Notable analytical errors generally arise from challenges in measuring pesticides at trace concentrations; variability in reagents, personnel, and protocols can amplify cumulative error unless appropriately managed. In this aspect, variance results from both G x E and analytical errors.

Accuracy and Precision of Measurements

Understanding Measurement Error

  • Every measurement has an inherent level of error:

    • A ruler allows measurements accurate to ± 0.5 mm.

    • A three-place balance measures accuracy to ± 0.0005 g.

    • A pH meter can measure to 0.1 decimal places (dp), thus being accurate to 0.05 dp.

  • It is crucial never to report results to more decimal places than certainty allows.

Defining Accuracy and Precision

  • Accuracy: Refers to how closely the mean of measurements aligns with the actual or 'true' value.

  • Precision: Concerns how repeatable or consistent the replicate samples are with one another.

  • Both accuracy and precision are distinct concepts; however, each is valued in experimental settings.

Difference Between Accuracy and Precision

  • Standard errors/deviations can be computed to reflect spread, which correlates with precision.

  • Achieving the true value is notoriously challenging; reference materials provide insights regarding the possible discrepancies in obtained results.

Experimental Objectives

Defining Questions Before Experiments

  • Clearly articulate what is being characterized in the research.

  • Validate the measurements to confirm they accurately reflect the target variables.

  • Ensure the values align with existing published data or analytical standards for comparability.

Challenges in Environmental and Biological Analyses

  • Low concentrations / densities within challenging media can hinder consistent locating of the analyte or organism.

  • Analytical procedures are often empirical, leading to variations based on methodology in sampling, preparation, and analysis.

  • Establishing quality control procedures is vital to instill confidence in results.

Coefficient of Variation and Its Implications

  • The coefficient of variation (%) is a critical statistic that assesses relative variability. Its presentation includes varying levels of precision at different concentrations, highlighting discrepancies across pharmaceuticals, pesticides, nutrients, etc.

Method Impact on Measurement

An example demonstrating the influence of methodology on measurement results, showcasing differences across tests (e.g., Dutch EA vs. USEPA) in sample testing for total cyanide in environments (groundwater and soil).

Data Utility and Measurement Requirements

Questions to Consider

  • Assess the required accuracy: Is absolute or relative value needed? Are standards necessary?

  • Evaluate the precision necessity: Where does precision matter most?

  • The balance of improvements in accuracy and precision against the sampling efficiency and resources available for the analytical agenda must be carefully considered.

Managing False Positives

Understanding Types of Errors

  • False Positives: Often arise from contamination and involve a statistical type I error rate of 5% with each test run.

  • Potential sources include sample contamination or interferences resulting from analytical techniques.

Contamination Control Strategies

  • Ensuring reagent tests are conducted under conditions where purity, contamination, and concentration can be verified is essential.

  • Implement laboratory blanks to manage apparatus contamination and use trip/field blanks to assess for storage artifacts.

  • Incorporate reference materials and samples for comparative analysis.

Addressing False Negatives

Identification of Type II Errors

  • A false negative occurs when a test fails to signify a real difference owing to factors such as insufficient sample size leading to poor recovery of analyte from tests or losses from analytes during storage.

  • Techniques that are not sensitive enough may lead to erroneous conclusions about the absence of statistically significant results.

Prevention Measures for False Negatives

  • Quality control measures can be employed, such as spiking sub-samples with known quantities of the analyte at different stages to gauge recovery rates.

  • Engaging reference materials also supports validation pipelines, and enhancement of methods to heighten sensitivity may be employed.

Importance of Replicates in Experiments

Capturing Variance

  • Replications are crucial to measuring and segmenting variance within experimental data. Understanding the errors from collection, measurement, storage, processing, and random variations between individuals is key to determining acceptable levels of variance.

  • Ultimately, remaining variance will be biological, associated with treatment or population effects, necessitating careful control and understanding.

Technical Replicates Defined

  • In experiments where the concentration of a chemical in a treated subject needs assessment, each sample may be processed in triplicate to measure variance tied to non-treatment parameters.

Handling Replicate Results

  • If measurements from three assays on a single sample align, the average is calculated for reporting.

  • In cases of outliers, consider discarding and possibly rerunning the analysis; otherwise, use the average of the two remaining measurements, refraining from employing all three for statistical analysis.

Capturing Biological Variance

  • Replications can also involve repeating experiments across different subjects to identify consistent biological responses, which may lead to insights influenced by cyclic biological processes or random external factors.

  • Whether multiple observations per subject are appropriate relies on the overarching experimental design and should provide a nuanced understanding of the system in question.

Summary of Upcoming Topics

  • In the following lecture, the focus will shift toward sample replication strategies, elucidating their purpose and essentiality in capturing true biological variance, alongside discussing statistical terms associated with error and the minimum sample numbers necessary for revealing true positive results.