Comprehensive Study Notes on Statistics and Data Handling in Analytical Chemistry

Fundamentals of Statistics and Data Handling in Analytical Chemistry

• Role and Necessity of Statistics in Analytical Chemistry:

  • Objective Mathematical Evaluation: Statistical methods provide objective mathematical tools required to extract the maximum amount of reliable information from finite experimental data sets.

  • Standardized Communication: Scientists require a universally recognized, standard format to communicate the significance, reliability, and precision of numerical experimental findings.

  • Experimental Design: Statistical principles establish the mathematical basis for optimal experimental design, minimizing effort and resource expenditure while maximizing precision and reliability.

  • Uncertainty Quantification: Replaces qualitative subjective descriptors such as "pretty sure," "very sure," "most likely," or "improbable" with rigorous numerical probability statements.

• Essential Statistical Terms and Definitions:

  • Population: A collection comprising an infinite or theoretically complete number of measurements ($N \rightarrow \infty$).

  • Sample: A finite set of experimental measurements drawn from and intended to represent the broader population.

  • True Value or True Mean ($\mu$): The theoretical, exact mean value of a given measurement if the entire infinite population were analyzed ($N \rightarrow \infty$).

  • Observed Mean ($\bar{x}$): The arithmetic average of a finite sample set of measurements, representing the central tendency of that experimental sample.

  • Average / Arithmetic Mean: The summation of all individual numerical measurements ($\sum x_i$) divided by the total count of measurements ($N$):     $\bar{x} = \frac{\sum_{i=1}^{N} x_i}{N}$

  • Median: The physical mid-point of a ranked data set arranged in ascending or descending numerical order.

  • In a data set with an odd count of observations, the median is the single value located exactly in the center.

  • In a data set with an even count of observations, the median is calculated as the arithmetic average of the central pair of numbers.

• Significant Figures:

  • Formal Definition: The number of digits necessary to express the results of a measurement consistent with the measured experimental precision, consisting of all digits known with absolute certainty plus exactly one final digit that is uncertain.

  • General Conventions:

  • All non-zero digits are always significant.

  • Any zeros situated between non-zero significant digits (captive zeros) are significant.

  • Final zeros or trailing zeros appearing in the decimal portion only are significant.

  • Addition and Subtraction Rule:

  • The resulting answer cannot possess more decimal places than the measurement having the smallest number of decimal places.

  • Example: Summing $43.7\,\text{g} + 4.432\,\text{g} + 12.68\,\text{g} = 60.812\,\text{g}$. Because $43.7\,\text{g}$ contains only one decimal place (tenths position), the final result must be reported as $60.8\,\text{g}$.

  • Multiplication and Division Rule (Relative Precision):

  • The least number of significant figures present in any term or operator determines the final number of significant figures in the calculated quotient or product.

  • Example: Multiplying $2.098 \times 3.23$. The raw calculator output displays $6.77654$. Because $3.23$ possesses only $3$ significant figures (whereas $2.098$ possesses $4$), the final answer must be rounded to $3$ significant figures: $6.78$.

Measurement Errors, Accuracy, Precision, and Uncertainty

• The Myth of "Error-Free Analysis":

  • Complete elimination of experimental error is impossible in scientific measurement.

  • Experimental errors can only be minimized through rigorous protocol adherence and approximated to an acceptable, quantifiable level of precision.

• Accuracy versus Precision:

  • Accuracy:

  • Theoretical definition: The degree of agreement between an experimental measured value and the absolute true value ($\mu$).

  • Realistic definition: Because the absolute true value is seldom known with ultimate perfection, accuracy is realistically defined as the degree of agreement between a measured value and an accepted true or certified reference value.

  • Precision:

  • The degree of agreement among replicate measurements performed on the exact same quantity under identical experimental conditions; it represents the repeatability or reproducibility of an analytical result.

  • Expressed quantitatively through parameters such as standard deviation ($s$), coefficient of variation ($\text{CV}$), range ($w$), and confidence interval of the mean.

  • Relationship Between Accuracy and Precision:

  • An analyst cannot attain true accuracy without good precision.

  • High precision does not guarantee accuracy; a precise set of data can suffer from large determinate (systematic) errors that displace all values simultaneously from the true value.

• Determinate (Systematic) Errors:

  • Characteristics: Determinate errors have definite values, an assignable cause, and are unidirectional (bias in one direction). They are determinable and can either be avoided or corrected.

  • Common Manifestations: Constant errors, uncalibrated balance weights, volumetric ware inaccuracies (such as buret volume misreadings), and variation in solution concentration induced by temperature changes.

  • Three Major Classifications of Determinate Errors:

  • Instrumental Error: Arises from faulty equipment, uncalibrated instruments, or deteriorated components. Overcome by careful, periodic recalibration and preventative maintenance of equipment.

  • Operative Error (Personal Error): Arises from lack of laboratory experience, carelessness, or personal bias/prejudice in reading graduations and recording measurements. Overcome by rigorous care, personal self-discipline, and objective technique.

  • Method Error: The most serious classification of analytical error, arising from non-ideal physical or chemical behavior of reagents, incomplete reactions, side reactions, or chemical instability. Overcome by running reagent blanks, analyzing Standard Reference Materials (SRM) with certified values, and validating results against two or more independent analytical methods.

• Indeterminate (Random or Accidental) Errors:

  • Characteristics: Arise from the limited ability of the analyst to control or correct subtle external conditions and an inability to detect or isolate minuscule fluctuations during observation.

  • Reflects inherent experimental uncertainty present in any physical measurement, revealed as small, random deviations among replicate readings.

  • Gaussian (Normal) Distribution:

  • In the absence of systematic errors, random errors strictly adhere to a Gaussian bell-shaped distribution curve.

  • Within an infinite population ($N \rightarrow \infty$), the probability that a random measurement falls within defined standard deviations ($\sigma$) from the true population mean ($\mu$) is:

    • Within $\pm 1\sigma$: $68\%$ of all observations.

    • Within $\pm 2\sigma$: $95\%$ of all observations (analysts are $95\%$ certain the true value lies within $\pm 2\sigma$ if no systematic error exists).

    • Within $\pm 3\sigma$: $99.7\%$ of all observations.

• Mathematical Methods for Expressing Accuracy:

  • Absolute Error ($E$):     $E = O - A$

  • Where $E$ is absolute error, $O$ is the observed experimental value, and $A$ is the accepted true value.

  • Worked Example: A calcium analysis yields an observed value of $18.24\%$, and the true certified value is $18.30\%$.       $E = 18.24\% - 18.30\% = -0.06\%$

  • Mean Error: The difference between the accepted true value and the experimental arithmetic mean value ($\bar{x} - A$).

  • Relative Error: The absolute error or mean error expressed as a percentage or fraction of the accepted true value:     $\text{Relative Error} = \left( \frac{\text{Absolute Error}}{\text{Accepted Value}} \right) \times 100\%$

  • Worked Example: With an absolute error of $-0.06\%$ and an accepted value of $18.30\%$       $\text{Relative Error} = \left( \frac{-0.06\%}{18.30\%} \right) \times 100\% = -0.33\%$       $\text{Relative Error (in parts per thousand, ppt)} = -0.33\% \times 10 = -3.3\,\text{ppt}$

  • Relative Accuracy: The observed value or experimental mean expressed as a percentage of the true value ($m$ or $\mu$):     $\%\,\text{Relative Accuracy} = \left( \frac{x}{\mu} \right) \times 100\%$

• Mathematical Methods for Expressing Precision:

  • Deviation from the Mean: The numerical difference between an individual measured value ($x_i$) and the arithmetic mean ($\bar{x}$):     $|x_i - \bar{x}|$

  • Deviation from the Median: The numerical difference between an individual measured value ($x_i$) and the median value.

  • Range ($w$): The numerical difference between the single highest (maximum) value and the single lowest (minimum) value in an experimental data set:     $w = x_{\text{highest}} - x_{\text{lowest}}$

  • Population Standard Deviation ($\sigma$ or $s_{\infty}$):

  • Applied when the data set represents an infinite population of experimental observations ($N \rightarrow \infty$):       $s = \sqrt{\frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}}$

    • Where $x_i$ is an individual measurement, $\mu$ is the true mean of the infinite population, and $N$ is the number of measurements.

  • Sample Standard Deviation ($s$):

  • Applied when analyzing a finite data set ($N < 30$), using degrees of freedom ($N - 1$) in the denominator to remove bias:       $s = \sqrt{\frac{\sum_{i=1}^{N} (x_i - \bar{x})^2}{N - 1}}$

  • Sample Variance ($s^2$): The square of the sample standard deviation:     $s^2 = \frac{\sum_{i=1}^{N} (x_i - \bar{x})^2}{N - 1}$

  • Standard Deviation of the Mean ($s_{\text{mean}}$): Quantifies the precision of the mean of $N$ measurements:     $s_{\text{mean}} = \frac{s}{\sqrt{N}}$

  • Relative Standard Deviation ($\text{RSD}$ or $s_r$):     $\text{RSD} = \frac{s}{\bar{x}}$

  • RSD in Parts Per Thousand (ppt):     $\text{RSD}_{\text{ppt}} = \left( \frac{s}{\bar{x}} \right) \times 1000\,\text{ppt}$

  • Coefficient of Variation ($\text{CV}$ or \%RSD):     $\text{CV} = \left( \frac{s}{\bar{x}} \right) \times 100\%$

• Distinction: Error versus Uncertainty:

  • Error: The quantitative difference between the measured value and the true value, serving as a measure of bias in a result.

  • Uncertainty: The quantification of the doubt surrounding the validity of a measurement result, characterizing the physical dispersion or range of possible values within which the measurement is expected to lie.

• Comprehensive Worked Exercise: Zinc Determination in Hair Samples:

  • Raw Replicate Data ($N = 5$): Sample measurements of zinc concentration ($\text{ppm Zn}$):

  • Sample 1: $x_1 = 0.752$, $x_1^2 = 0.565504$

  • Sample 2: $x_2 = 0.756$, $x_2^2 = 0.571536$

  • Sample 3: $x_3 = 0.752$, $x_3^2 = 0.565504$

  • Sample 4: $x_4 = 0.751$, $x_4^2 = 0.564001$

  • Sample 5: $x_5 = 0.760$, $x_5^2 = 0.577600$

  • Intermediate Summations:     $\sum x_i = 3.771$     $\sum x_i^2 = 2.8440882$

  • Step-by-Step Calculations:

  • a. Mean of the data ($\bar{x}$):       $\bar{x} = \frac{\sum x_i}{N} = \frac{3.771}{5} = 0.7542\,\text{ppm} \approx 0.754\,\text{ppm Zn}$

  • b. Standard Deviation ($s$):       $s = \sqrt{\frac{\sum (x_i - \bar{x})^2}{N - 1}} = 0.00377\,\text{ppm} \approx 0.0038\,\text{ppm Zn}$

  • c. Variance ($s^2$):       $s^2 = (0.0038)^2 = 1.4 \times 10^{-5}$

  • d. Relative Standard Deviation ($\text{RSD}$, $s_r$):       $s_r = \frac{s}{\bar{x}} = \frac{0.0038}{0.754} = 5.0 \times 10^{-3}$

  • e. Relative Standard Deviation in ppt:       $\text{RSD}_{\text{ppt}} = s_r \times 1000 = 5.0 \times 10^{-3} \times 1000 = 5.0\,\text{ppt}$

  • f. Coefficient of Variation (\%RSD / CV):       $\text{CV} = \left( \frac{s}{\bar{x}} \right) \times 100\% = 0.50\%$

  • g. Range ($w$):       $w = x_{\text{largest}} - x_{\text{smallest}} = 0.760 - 0.751 = 0.009\,\text{ppm Zn}$

Confidence Intervals and Limits

• Core Conceptual Framework:

  • Because the true population mean ($\mu$) cannot be determined directly, statistical theory is used to establish numerical boundaries around the experimental mean ($\bar{x}$) that have a stated probability of containing $\mu$.

  • Confidence Interval (CI): The physical numerical range within which the true value $\mu$ is estimated to fall ($a < \bar{x} < b$).

  • Confidence Limit (CL): The extreme boundaries defining the confidence interval magnitude ($\bar{x} \pm \text{confidence interval}$).

  • Confidence Level: The chosen probability (expressed as a percentage) that the true population mean lies within the defined limits (common levels: $50\%$, $68\%$, $80\%$, $90\%$, $95\%$, $99\%$, and $99.7\%$).

• Governing Mathematical Equation for Confidence Limits:   $\text{Confidence Limit} = \bar{x} \pm \frac{ts}{\sqrt{N}}$

  • Where $\bar{x}$ is the experimental sample mean, $s$ is the sample standard deviation, $N$ is the total number of replicate analyses, and $t$ is the Student's $t$-value determined by degrees of freedom ($\nu = N - 1$) and the designated confidence level.

• Selecting an Appropriate Confidence Level:

  • Too High a Confidence Level (e.g., $> 99.5\%$): Generates an excessively broad interval that may erroneously encompass non-random fluctuations or outliers.

  • Too Low a Confidence Level (e.g., $< 80\%$): Generates an overly narrow interval that inappropriately excludes valid random experimental points.

  • A confidence level of $90\%$ to $95\%$ ($95\%$ is standard) is universally regarded as reasonable in analytical practice.

• Reference Values of Student's $t$ across Confidence Levels:

  • Degrees of Freedom ($ u = 1$): $80\% = 3.08$, $90\% = 6.314$, $95\% = 12.706$, $99\% = 63.657$, $99.5\% = 127.32$, $99.9\% = 637$

  • Degrees of Freedom ($ u = 2$): $80\% = 1.89$, $90\% = 2.920$, $95\% = 4.303$, $99\% = 9.925$, $99.5\% = 14.089$, $99.9\% = 31.6$

  • Degrees of Freedom ($ u = 3$): $80\% = 1.64$, $90\% = 2.353$, $95\% = 3.182$, $99\% = 5.841$, $99.5\% = 7.453$, $99.9\% = 12.9$

  • Degrees of Freedom ($ u = 4$): $80\% = 1.53$, $90\% = 2.132$, $95\% = 2.776$, $99\% = 4.604$, $99.5\% = 5.598$, $99.9\% = 8.60$

  • Degrees of Freedom ($ u = 5$): $80\% = 1.48$, $90\% = 2.015$, $95\% = 2.571$, $99\% = 4.032$, $99.5\% = 4.773$, $99.9\% = 6.86$

  • Degrees of Freedom ($ u = 6$): $80\% = 1.44$, $90\% = 1.943$, $95\% = 2.447$, $99\% = 3.707$, $99.5\% = 4.317$, $99.9\% = 5.96$

  • Degrees of Freedom ($ u = 7$): $80\% = 1.42$, $90\% = 1.895$, $95\% = 2.365$, $99\% = 3.500$, $99.5\% = 4.029$, $99.9\% = 5.40$

  • Degrees of Freedom ($ u = 8$): $80\% = 1.40$, $90\% = 1.860$, $95\% = 2.306$, $99\% = 3.355$, $99.5\% = 3.832$, $99.9\% = 5.04$

  • Degrees of Freedom ($ u = 9$): $80\% = 1.38$, $90\% = 1.833$, $95\% = 2.262$, $99\% = 3.250$, $99.5\% = 3.690$, $99.9\% = 4.78$

  • Degrees of Freedom ($ u = 10$): $80\% = 1.36$, $90\% = 1.812$, $95\% = 2.228$, $99\% = 3.169$, $99.5\% = 3.581$, $99.9\% = 4.59$

  • Degrees of Freedom ($ u = 15$): $90\% = 1.753$, $95\% = 2.131$, $99\% = 2.947$, $99.5\% = 3.252$

  • Degrees of Freedom ($ u = 20$): $90\% = 1.725$, $95\% = 2.086$, $99\% = 2.845$, $99.5\% = 3.153$

  • Degrees of Freedom ($ u = 25$): $90\% = 1.708$, $95\% = 2.060$, $99\% = 2.787$, $99.5\% = 3.078$

  • Degrees of Freedom ($u = \infty$): $80\% = 1.28$, $90\% = 1.645$, $95\% = 1.960$, $99\% = 2.576$, $99.5\% = 2.807$, $99.9\% = 3.29$

• Confidence Limit Worked Examples:

  • Example 1: Calcium Analysis:

  • Data set ($N = 5$): $14.35\%$, $14.41\%$, $14.40\%$, $14.32\%$, and $14.37\%$.

  • Mean: $\bar{x} = 14.37\%$

  • Standard deviation: $s = 0.037\%$

  • Degrees of freedom: $\nu = 5 - 1 = 4$

  • Critical $t$-value at $95\%$ confidence level ($\nu = 4$): $t = 2.776$

  • Confidence Limit Calculation:       $\text{CL} = \bar{x} \pm \frac{ts}{\sqrt{N}} = 14.37\% \pm \frac{2.776 \times 0.037\%}{\sqrt{5}} = 14.37\% \pm 0.05\%$

  • Example 2: Soda Ash ($\text{Na}_2\text{CO}_3$) Titration:

  • Triplicate analysis ($N = 3$): $93.50\%$, $93.58\%$, and $93.43\,\text{wt\%}$.

  • Mean: $\bar{x} = 93.50\%$

  • Standard deviation: $s = 0.075\%$

  • Degrees of freedom: $\nu = 3 - 1 = 2$

  • Critical $t$-value at $95\%$ confidence level ($\nu = 2$): $t = 4.303$

  • Confidence Interval Calculation:       $\text{CI} = \bar{x} \pm \frac{ts}{\sqrt{N}} = 93.50\% \pm \frac{4.303 \times 0.075\%}{\sqrt{3}} = 93.50\% \pm 0.19\%$

  • Example 3: Blood Alcohol Content ($\text{\%C}_2\text{H}_5\text{OH}$):

  • Blood analysis results ($N = 3$): $0.084\%$, $0.089\%$, and $0.079\%$

  • Intermediate values: $\sum x_i = 0.252$, $\sum x_i^2 = (0.084)^2 + (0.089)^2 + (0.079)^2 = 0.007056 + 0.007921 + 0.006241 = 0.021218$

  • Mean: $\bar{x} = \frac{0.252}{3} = 0.084\%$

  • Standard deviation:       $s = \sqrt{\frac{0.021218 - \frac{(0.252)^2}{3}}{3 - 1}} = 0.0050\%\,\text{C}_2\text{H}_5\text{OH}$

  • Degrees of freedom: $\nu = 3 - 1 = 2$

  • Critical $t$-value ($95\%$, $\nu = 2$): $t = 4.30$

  • Calculation:       $95\%\,\text{CI} = \bar{x} \pm \frac{ts}{\sqrt{N}} = 0.084\% \pm \frac{4.30 \times 0.0050\%}{\sqrt{3}} = 0.084\% \pm 0.012\%\,\text{C}_2\text{H}_5\text{OH}$

• Applications of Confidence Intervals:

  • Determination of Required Replicates: To establish the exact number of experimental replicates ($N$) necessary to narrow the confidence interval to an analytically desired threshold ($|\mu - \bar{x}|$):     $N = \left( \frac{ts}{|\mu - \bar{x}|} \right)^2$

  • Worked Example: Reducing Confidence Interval to Specified Margin:

  • Goal: Reduce confidence interval to $1.5\,\mu\text{g/mL}$ at $95\%$ confidence level given $s = 2.4\,\mu\text{g/mL}$.

  • For large/infinite population approximation, the critical $t$-value at $95\%$ is $t = 1.96$.

  • Calculation:       $N = \left( \frac{ts}{|\mu - \bar{x}|} \right)^2 = \left( \frac{1.96 \times 2.4}{1.5} \right)^2 = 9.83 \approx 10\,\text{replicates}$

  • Worked Example: Copper Determination in Fuel by Atomic Absorption:

  • Parameters: Pooled standard deviation $s_{\text{pooled}} = 0.27\,\mu\text{g Cu/mL}$. Desired $95\%$ confidence limit threshold $= 0.20\,\mu\text{g Cu/mL}$.

  • Relationship:       $0.20 = \frac{1.96 \times 0.27}{\sqrt{N}}$

  • Solving for $N$:       $N = \left( \frac{1.96 \times 0.27}{0.20} \right)^2 = 7.00$

  • Exactly $N = 7$ replicate measurements are required.

Hypothesis Testing and Significance Tests

• Principles of Hypothesis Testing:

  • Advanced to explain experimental observations: a hypothetical mathematical model is established and tested experimentally to establish its physical validity.

  • Null Hypothesis ($H_0$): Assumes that numerical quantities under comparison (means or variances) are identical, and that observed experimental discrepancies are due exclusively to indeterminate random variations.

  • Decision Criterion:

  • The calculated test parameter ($\text{Parameter}_{\text{calculated}}$) derived from experimental data is compared against the critical theoretical parameter from standard tables ($\text{Parameter}_{\text{table}}$).

  • If $\text{Parameter}_{\text{calculated}} \le \text{Parameter}_{\text{table}}$, the null hypothesis $H_0$ is accepted (differences are not statistically significant).

  • If $\text{Parameter}_{\text{calculated}} > \text{Parameter}_{\text{table}}$, the null hypothesis $H_0$ is rejected (differences are statistically significant at that confidence level).

• The $F$-Test for Comparing Variances (Precision):

  • Purpose: Designed to determine whether two analytical methods differ significantly in their precision (variances).

  • Mathematical Formula:     $F = \frac{s_1^2}{s_2^2}$

  • The larger variance is placed in the numerator so that $F_{\text{calculated}} \ge 1$.

  • Degrees of freedom are $\nu_1 = N_1 - 1$ (numerator) and $\nu_2 = N_2 - 1$ (denominator).

  • Decision Rule: If $F_{\text{calculated}} > F_{\text{table}}$, a statistically significant difference exists in precision between the two methods at the $95\%$ confidence level.

• $F$-Test Worked Example: Glucose Determination Method Comparison:

  • Context: A newly developed colorimetric procedure is compared against the standard Folin-Wu method for determining blood serum glucose concentration.

  • Data:

  • New Method ($\text{mg/dL}$): $127, 125, 123, 130, 131, 126, 129$

    • $N_1 = 7$, Mean $\bar{x}_1 = 127\,\text{mg/dL}$

    • Variance calculation:         $s_1^2 = \frac{\sum (x_{i1} - \bar{x}_1)^2}{N_1 - 1} = \frac{50}{7 - 1} = 8.3$

  • Folin-Wu Method ($\text{mg/dL}$): $130, 128, 131, 129, 127, 125$

    • $N_2 = 6$, Mean $\bar{x}_2 = 128\,\text{mg/dL}$

    • Variance calculation:         $s_2^2 = \frac{\sum (x_{i2} - \bar{x}_2)^2}{N_2 - 1} = \frac{24}{6 - 1} = 4.8$

  • $F$-Ratio Calculation:     $F = \frac{8.3}{4.8} = 1.73$

  • Statistical Conclusion: Tabulated critical $F$-value for $\nu_1 = 6$ and $\nu_2 = 5$ at the $95\%$ confidence level is $4.95$. Because $F_{\text{calculated}} (1.73) < F_{\text{table}} (4.95)$, $H_0$ is accepted: there is no statistically significant difference in precision between the two methods.

• The Student's $t$-Test:

  • Used to determine whether a significant difference exists between experimental means or between an experimental mean and a certified reference standard.

  • Evaluated under three distinct experimental circumstances:

  1. Experimental mean compared to a certified/known reference value.

  2. Comparison of the experimental means of two independent analytical methods.

  3. Paired $t$-test on a series of differing samples analyzed by two distinct methods.

• Case 1: True / Certified Reference Value is Known:

  • Goals: To evaluate whether a newly developed method departs from a certified reference standard, or to determine whether a measured set of values exceeds a prescribed regulatory limit.

  • Mathematical Formula:     $\pm t = \frac{|\bar{x} - \mu|}{s}\sqrt{N}$

  • Where $\bar{x}$ is the experimental mean, $\mu$ is the certified reference value, $s$ is the sample standard deviation, and $N$ is the sample count.

  • Worked Example: Five replicate analyses ($N = 5$) yield a mean of $10.8\,\text{ppm}$ with $s = 0.7\,\text{ppm}$. The certified reference value is $11.7\,\text{ppm}$.

  • Degrees of freedom: $\nu = N - 1 = 5 - 1 = 4$

  • Calculation:       $t_{\text{calc}} = \frac{|10.8 - 11.7|}{0.7}\sqrt{5} = \frac{0.9 \times 2.236}{0.7} = 2.9$

  • Tabulated $t$-value at the $95\%$ confidence level for $\nu = 4$ is $2.776$.

  • Decision: Because $t_{\text{calc}} (2.9) > t_{\text{table}} (2.776)$, the null hypothesis $H_0$ is rejected. Analysts are $95\%$ confident that the developed procedure produces a value statistically different from the certified true value.

• Case 2: Comparison of the Means of Two Independent Methods:

  • Prerequisite: Before applying this test, both methods must have statistically identical precision (verified via the $F$-test).

  • Pooled Standard Deviation ($s_p$): Combines individual sample deviations to calculate an improved estimate of precision across multiple sample sets ($k$ samples):     $s_p = \sqrt{\frac{\sum_{i=1}^{N_1}(x_{i1} - \bar{x}_1)^2 + \sum_{i=1}^{N_2}(x_{i2} - \bar{x}_2)^2 + \dots + \sum_{i=1}^{N_k}(x_{ik} - \bar{x}_k)^2}{N - k}}$

  • Where $N = N_1 + N_2 + \dots + N_k$, and $k$ is the total number of distinct sample groups.

  • $t$-Test Formula:     $\pm t = \frac{|\bar{x} - \bar{y}|}{s_p}\sqrt{\frac{M N}{M + N}}$

  • Where $\bar{x}$ is the mean of $M$ replicate measurements (Method 1), $\bar{y}$ is the mean of $N$ replicate measurements (Method 2), and degrees of freedom $\nu = M + N - 2$.

  • Worked Example: Iron in Ore Determination:

  • Test Method ($N_1 = 6$): $20.10\%$, $20.50\%$, $18.65\%$, $19.25\%$, $19.40\%$, $19.99\%$. Mean $\bar{x}_1 = 19.65\%$.

    • Individual deviations $(x_{i1} - \bar{x}_1)$: $0.45, 0.85, -1.00, -0.40, -0.25, 0.34$

    • Sum of squared deviations: $\sum (x_{i1} - \bar{x}_1)^2 = 0.202 + 0.722 + 1.000 + 0.160 + 0.062 + 0.116 = 2.262$

  • Reference Method ($N_2 = 5$): $18.89\%$, $19.20\%$, $19.00\%$, $19.70\%$, $19.40\%$. Mean $\bar{x}_2 = 19.24\%$.

    • Individual deviations $(x_{i2} - \bar{x}_2)$: $-0.35, -0.04, -0.24, 0.46, 0.16$

    • Sum of squared deviations: $\sum (x_{i2} - \bar{x}_2)^2 = 0.122 + 0.002 + 0.058 + 0.212 + 0.026 = 0.420$

  • Preliminary $F$-Test:       $F = \frac{s_1^2}{s_2^2} = \frac{2.262 / 5}{0.420 / 4} = \frac{0.4524}{0.1050} = 4.31$

    • Tabulated critical $F$ ($\nu_1 = 5$, $\nu_2 = 4$ at $95\%$) is $6.26$. Since $4.31 < 6.26$, the standard deviations are statistically indistinguishable; pooling is valid.

  • Calculation of Pooled Standard Deviation ($s_p$):       $s_p = \sqrt{\frac{2.262 + 0.420}{6 + 5 - 2}} = \sqrt{\frac{2.682}{9}} = 0.546$

  • Calculation of $t_{\text{calculated}}$:       $\pm t = \frac{19.65 - 19.24}{0.546}\sqrt{\frac{6 \times 5}{6 + 5}} = \frac{0.41}{0.546}\sqrt{\frac{30}{11}} = 1.23$

  • Decision: Degrees of freedom $\nu = 6 + 5 - 2 = 9$. Tabulated $t$ at the $95\%$ confidence level is $2.262$. Because $t_{\text{calc}} (1.23) < t_{\text{table}} (2.262)$, $H_0$ is accepted: no significant difference exists between the two analytical methods.

• Case 3: Paired $t$-Test (Series of Different Samples):

  • Application: Used when analyzing a series of different natural samples spanning varying concentrations using two distinct analytical methods, where sample composition variability precludes direct pooling.

  • Mathematical Formulation:     $t = \frac{|\bar{D}|}{s_d}\sqrt{N}$     $s_d = \sqrt{\frac{\sum_{i=1}^{N} (D_i - \bar{D})^2}{N - 1}}$

  • Where $D_i$ represents the individual difference between the two analytical methods for sample $i$, $\bar{D}$ is the average difference across all $N$ samples, and $s_d$ is the sample standard deviation of the differences.

  • Worked Example: Comparison across six distinct serum samples ($A$ through $F$):

  • Sample A: Your Method = $10.2\,\text{mg/dL}$, Standard = $10.5\,\text{mg/dL}$, $D_i = -0.3$, $D_i - \bar{D} = -0.6$, $(D_i - \bar{D})^2 = 0.36$

  • Sample B: Your Method = $12.7\,\text{mg/dL}$, Standard = $11.9\,\text{mg/dL}$, $D_i = 0.8$, $D_i - \bar{D} = 0.5$, $(D_i - \bar{D})^2 = 0.25$

  • Sample C: Your Method = $8.6\,\text{mg/dL}$, Standard = $8.7\,\text{mg/dL}$, $D_i = -0.1$, $D_i - \bar{D} = -0.4$, $(D_i - \bar{D})^2 = 0.16$

  • Sample D: Your Method = $17.5\,\text{mg/dL}$, Standard = $16.9\,\text{mg/dL}$, $D_i = 0.6$, $D_i - \bar{D} = 0.3$, $(D_i - \bar{D})^2 = 0.09$

  • Sample E: Your Method = $11.2\,\text{mg/dL}$, Standard = $10.9\,\text{mg/dL}$, $D_i = 0.3$, $D_i - \bar{D} = 0.0$, $(D_i - \bar{D})^2 = 0.00$

  • Sample F: Your Method = $11.5\,\text{mg/dL}$, Standard = $11.1\,\text{mg/dL}$, $D_i = 0.4$, $D_i - \bar{D} = 0.1$, $(D_i - \bar{D})^2 = 0.01$

  • Summations: $\sum D_i = 1.7$, $\bar{D} = \frac{1.7}{6} = 0.283 \approx 0.28$, $\sum (D_i - \bar{D})^2 = 0.87$

  • Calculation of $s_d$ and $t$:       $s_d = \sqrt{\frac{0.87}{6 - 1}} = 0.42$       $t = \frac{0.28}{0.42}\sqrt{6} = 1.63$

  • Decision: Degrees of freedom $\nu = 6 - 1 = 5$. Tabulated $t$ at $95\%$ confidence level is $2.571$. Because $t_{\text{calc}} (1.63) < t_{\text{table}} (2.571)$, $H_0$ is accepted: no significant difference exists between the two procedures.

Rejection of Outliers: The Dixon Q-Test

• Purpose and Fundamentals:

  • Applied to statistically determine whether an apparent outlier (a questionable data point) should be rejected due to a determinate error, or retained because it falls within the expected range of random indeterminate variation.

• Procedural Steps for Calculating $Q$:

  1. Arrange the experimental data in strictly ascending numerical order.

  2. Compute the total experimental range ($w$), defined as the absolute difference between the highest and lowest values:      $w = x_{\text{highest}} - x_{\text{lowest}}$

  3. Determine the gap ($a$), defined as the absolute difference between the suspect value and its nearest numerical neighbor:      $a = |x_{\text{suspect}} - x_{\text{nearest}}|$

  4. Calculate $Q_{\text{calc}}$:      $Q_{\text{calc}} = \frac{a}{w} = \frac{\text{outlier difference}}{\text{range}}$

  5. Compare $Q_{\text{calc}}$ against the critical rejection quotient ($Q_{\text{table}}$) at the desired confidence level.

  6. Rejection Rule: If $Q_{\text{calc}} \ge Q_{\text{table}}$, reject the outlier as being contaminated by a systematic/determinate error. If $Q_{\text{calc}} < Q_{\text{table}}$, retain the outlier as a valid manifestation of random error.

• Critical Rejection Quotients ($Q$-Table) at Defined Confidence Limits (Adapted from D. B. Rorabacher, 1991):

  • $N = 3$: $Q_{90} = 0.941$, $Q_{95} = 0.970$, $Q_{99} = 0.994$

  • $N = 4$: $Q_{90} = 0.765$, $Q_{95} = 0.829$, $Q_{99} = 0.926$

  • $N = 5$: $Q_{90} = 0.642$, $Q_{95} = 0.710$, $Q_{99} = 0.821$

  • $N = 6$: $Q_{90} = 0.560$, $Q_{95} = 0.625$, $Q_{99} = 0.740$

  • $N = 7$: $Q_{90} = 0.507$, $Q_{95} = 0.568$, $Q_{99} = 0.680$

  • $N = 8$: $Q_{90} = 0.468$, $Q_{95} = 0.526$, $Q_{99} = 0.634$

  • $N = 9$: $Q_{90} = 0.437$, $Q_{95} = 0.493$, $Q_{99} = 0.598$

  • $N = 10$: $Q_{90} = 0.412$, $Q_{95} = 0.466$, $Q_{99} = 0.568$

  • $N = 15$: $Q_{90} = 0.338$, $Q_{95} = 0.384$, $Q_{99} = 0.475$

  • $N = 20$: $Q_{90} = 0.300$, $Q_{95} = 0.342$, $Q_{99} = 0.425$

  • $N = 25$: $Q_{90} = 0.277$, $Q_{95} = 0.317$, $Q_{99} = 0.393$

  • $N = 30$: $Q_{90} = 0.260$, $Q_{95} = 0.298$, $Q_{99} = 0.372$

• $Q$-Test Worked Example: Chloride Determinations in Serum:

  • Experimental Data Set: Four replicate determinations of chloride in pooled serum: $103, 106, 107, 114\,\text{meq/L}$.

  • Assessment of Suspect Value: The value $114\,\text{meq/L}$ appears suspect.

  • Step-by-Step Calculation:

  • Gap ($a$): Difference from nearest neighbor ($107\,\text{meq/L}$):       $a = 114 - 107 = 7\,\text{meq/L}$

  • Range ($w$): Difference from lowest reading ($103\,\text{meq/L}$):       $w = 114 - 103 = 11\,\text{meq/L}$

  • Calculated $Q$:       $Q_{\text{calc}} = \frac{7}{11} = 0.64$

  • Tabulated Critical $Q$ ($N = 4$, $95\%$ confidence level): $Q_{\text{table}} = 0.829$

  • Statistical Decision: Because $Q_{\text{calc}} (0.64) < Q_{\text{table}} (0.829)$, the suspect value cannot be attributed to a determinate error; it falls within expected random error and must be retained.

Linear Regression and Calibration by Least Squares

• Method of Least Squares:

  • Used to determine the best-fit straight line ($y = mx + b$) for an analytical calibration curve.

  • Assumes that errors in the concentration values ($x$) are negligible, and that deviations reside predominantly in the detector response ($y$).

  • Minimizes the sum of the squares of the vertical deviations (the residuals) between experimental data points and the regression line.

• Governing Least Squares Equations:

  • Slope ($m$):     $m = \frac{\sum x_i y_i - \frac{(\sum x_i)(\sum y_i)}{n}}{\sum x_i^2 - \frac{(\sum x_i)^2}{n}} = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

  • Y-Intercept ($b$):     $b = \bar{y} - m\bar{x}$

• Worked Example: Fluorometric Determination of Riboflavin (Vitamin $B_2$):

  • Experimental Context: Riboflavin is determined in a cereal sample by measuring its fluorescence intensity in a $5\%$ acetic acid solution. A calibration curve is generated from standards, and an unknown sample produces a fluorescence intensity reading of $15.4\,\text{arbitrary units}$.

  • Calibration Standards Data Set ($n = 5$):

  • Standard 1: Concentration $x_1 = 0.000\,\mu\text{g/mL}$, Intensity $y_1 = 0.0$, $x_1^2 = 0.0000$, $x_1 y_1 = 0.00$

  • Standard 2: Concentration $x_2 = 0.100\,\mu\text{g/mL}$, Intensity $y_2 = 5.8$, $x_2^2 = 0.0100$, $x_2 y_2 = 0.58$

  • Standard 3: Concentration $x_3 = 0.200\,\mu\text{g/mL}$, Intensity $y_3 = 12.2$, $x_3^2 = 0.0400$, $x_3 y_3 = 2.44$

  • Standard 4: Concentration $x_4 = 0.400\,\mu\text{g/mL}$, Intensity $y_4 = 22.3$, $x_4^2 = 0.1600$, $x_4 y_4 = 8.92$

  • Standard 5: Concentration $x_5 = 0.800\,\mu\text{g/mL}$, Intensity $y_5 = 43.3$, $x_5^2 = 0.6400$, $x_5 y_5 = 34.64$

  • Summary Statistics:     $\sum x_i = 1.500$     $\sum y_i = 83.6$     $\sum x_i^2 = 0.8500$     $\sum x_i y_i = 46.58$     $(\sum x_i)^2 = 2.250$     $\bar{x} = \frac{\sum x_i}{n} = \frac{1.500}{5} = 0.3000$     $\bar{y} = \frac{\sum y_i}{n} = \frac{83.6}{5} = 16.72$

  • Slope Calculation ($m$):     $m = \frac{46.58 - \frac{1.500 \times 83.6}{5}}{0.8500 - \frac{2.250}{5}} = \frac{46.58 - 25.08}{0.8500 - 0.4500} = \frac{21.50}{0.4000} = 53.75\,\text{fluor. units/ppm}$

  • Intercept Calculation ($b$):     $b = 16.72 - (53.75 \times 0.3000) = 16.72 - 16.125 = 0.595 \approx 0.60\,\text{fluor. units}$

  • Final Calibrated Straight-Line Equation:     $y\,(\text{FU}) = 53.8\,(\text{FU/ppm})\,x\,(\text{ppm}) + 0.6\,(\text{FU})$

  • Sample Concentration Determination:     $15.4 = 53.8x + 0.6$     $53.8x = 14.8$     $x = 0.275\,\mu\text{g/mL}$

• Correlation Coefficient ($r$):

  • Definition: A standardized measure of the strength and direction of the linear relationship between two variables ($x$ and $y$).

  • Mathematical Formulation:     $r = \frac{\sum x_i y_i - n\bar{x}\bar{y}}{\sqrt{\left( \sum x_i^2 - n\bar{x}^2 \right)\left( \sum y_i^2 - n\bar{y}^2 \right)}}$     $r = \frac{n\sum x_i y_i - (\sum x_i)(\sum y_i)}{\sqrt{\left[ n\sum x_i^2 - (\sum x_i)^2 \right]\left[ n\sum y_i^2 - (\sum y_i)^2 \right]}}$

• Limit of Detection (LOD) and Limit of Quantitation (LOQ):

  • Limit of Detection (LOD):

  • The lowest analyte concentration level that can be reliably differentiated from an analytical reagent blank with statistical certainty.

  • Criterion: The analyte concentration that generates an instrument signal equal to the mean background blank signal plus three times the standard deviation of the blank:       $\text{Signal}_{\text{LOD}} = \bar{S}_{\text{blank}} + 3s_{\text{blank}}$

  • Limit of Quantitation (LOQ): Defines the lowest concentration of analyte in a sample that can be quantitatively determined with suitable precision and accuracy under stated operational conditions.