Foundations of Research Methods — Comprehensive Study Notes

What is Scientific Research?

  • Scientific research is the systematic investigation of scientific theories and hypotheses.
  • It is a process of rigorous reasoning based on interactions among theories, methods, and findings.
  • It builds on understanding derived from objective testing of models or theories.
  • The accumulation of scientific knowledge is laborious, plodding, circuitous, and indirect.
  • Scientific knowledge is developed and improved through critique, contested findings, replication, and convergence.
  • Scientific knowledge is developed through sustained efforts.
  • Scientific inquiry must be guided by fundamental principles.

Fundamental Principles

I. Ask significant questions that can be answered empirically

  • The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill. To raise new questions, new possibilities, to regard old questions from a new angle, requires creative imagination and marks real advance in science (Einstein & Infeld, 1938).
  • The research questions must be asked in a way that allows empirical investigation.

II. Link research to relevant theory

  • Scientific research can be guided by a conceptual framework, model, or theory that generates questions to be asked or answers to the questions posed.
  • Theory drives the research question, the use of methods, and the interpretation of results.

III. Select and apply research designs and methods that permit direct investigation of the question

  • Trustworthiness of a study is predicated on major elements:
    • Suitability of the proposed design or methodology to address the specific questions posed.
    • Scientific rigor by which the methodology is applied.
    • The link between question and methodology must be clear and justified.
    • Detailed description of the method, measures, data collection procedures, data analyses, and subjects must be available to permit replication.

IV. Provide a coherent and explicit chain of reasoning that can be replicated

  • What assumptions underlie the inferences, and were they clearly stated and justified?
  • How was evidence judged to be relevant?
  • How were alternative, competing hypotheses identified, considered, and accounted for (accepted or discarded)?

V. Replicate and generalize across studies

  • Internal Validity: Observations are consistent and generalizable across observers, tasks, and measurement occasions.
    • Statistical methods e.g., correlation;
    • Non-statistical methods e.g., triangulation, comparative analysis.
  • External Validity: Extent to which treatment conditions and participant population reflect the world to which generalization is desired.

VI. Report research publicly to encourage professional scrutiny, critique and replication

  • Criticism is essential to scientific progress.
  • The extent to which new findings can be reviewed, contested, and accepted or rejected depends on accurate, comprehensive, and accessible records of:
    • Data
    • Methods
    • Inferential reasoning

Common Conceptions/Misconceptions About Scientific Research

  • ✓ Experimental research is more "scientific" than descriptive or qualitative research.
  • A study is deemed to be "scientific" when:
    • There are clear, testable questions underlying the design.
    • The methods are appropriate to answer the questions and falsify competing hypotheses.
    • The study is explicitly linked to theory and previous research.
    • The data are analyzed systematically and with appropriate tools.
    • The data are made available for review and criticism.
  • ✓ Research in education is fundamentally different from the hard sciences.

Basic Elements of the Scientific Methods

  • Scientific method refers to a standardized set of techniques for building scientific knowledge, such as:
    • how to make valid observations,
    • how to interpret results,
    • and how to generalize those results.
  • 1. Replicability: Replicate or repeat a scientific study and obtain similar, if not identical, results.
  • 2. Precision: Theoretical concepts defined so that others can measure those concepts and test the theory.
  • 3. Falsifiability: Theories that cannot be tested or falsified are not scientific theories; such knowledge is not scientific knowledge.
  • 4. Parsimony: When there are multiple explanations, accept the simplest or most economical explanation.

Types of Scientific Research

  • Scientific Research categories: Exploratory, Descriptive, Explanatory.
  • 1. Quantitative Research: data are numeric, often associated with the positivist/post-positivist paradigm; aims to enable statistical calculations and generalizable conclusions.
  • 2. Qualitative Research: associated with the social constructivist paradigm; emphasizes the socially constructed nature of reality, aims to uncover deeper meanings and significance of human behavior and experience; seeks rich, comprehensive understanding rather than broad generalization.

Scientific Method of Research: Four Steps

  • 1. Observation
  • 2. Hypothesis
  • 3. Testing
  • 4. Predictions
  • The cycle involves Induction (from Observation to Theory) and Deduction (theory guiding observations).
  • If the experiments prove the hypothesis true, it becomes a theory or law of nature.
  • If the experiments prove the hypothesis false, the hypothesis must be rejected or modified.
  • Proper use of the method should yield predictive power for phenomena not yet tested.

Observation

  • An act of recognizing and noting a fact or occurrence, often involving measurement with instruments.
  • Experiments performed in the laboratory; experiments drawn from the literature.
  • Keys to implementation in research:
    • Sorting Observations (from Literature Searches):
    • Useful
    • Contains unanswered questions you think you can address
    • Not Useful (Yet!)
  • Formulate Hypothesis.
  • Note results (in report format—optional) for future use in confirming/denying your hypothesis once it is found.

Hypothesis

  • Hypothesis: a tentative assumption made to draw out and test its logical or empirical consequences.
  • Good Hypothesis can be TESTED with Experiment or Calculation.
  • Requires substantial thought and reading—leap from observation to hypothesis; worth the effort because it prepares you for laboratory work.
  • Suggestions for developing hypotheses:
    • Couple archival journal reading with theory.
    • Couple with common sense or intuition about how things should be in the system studied.
    • Follow logical reasoning about what you read with scientific and/or mathematical basis (use drawings/diagrams).
    • Write up your thoughts and opinions in report format or in a notebook.
  • Hypothesis shapes the study in terms of:
    • Identification of study sample size
    • Issues in data collection
    • Proper analysis of the data
    • Data interpretation
  • Traditional framing:
    • Ho: "Null" hypothesis (assumed)
    • H1: "Alternative" hypothesis
  • Examples:
    • Ho: There is no association between the exposure and disease of interest
    • H1: There is an association between the exposure and disease of interest (beyond what might be expected from random error alone)

Experimental Testing

  • To apply a test as a means of analysis or diagnosis.
  • Keys to implementation:
    • Good TESTS will prove or disprove your hypothesis.
    • Experimental tests can be performed within computing (e.g., Coventor model with calculations and predictions).
    • Consider all alternatives; an experiment may not disprove all aspects of the hypothesis, but may disprove parts; note which aspects the experiment tests.
    • Consider the availability of instrumentation to perform your tests.

Predictions

  • Definition: To declare or indicate in advance; especially foretell on the basis of observation, experience, or scientific reason.
  • Keys to implementation:
    • Good Predictions can be tested against your hypothesis.
    • Revisit basics (textbook theory) to develop a mathematical model/construct to help make predictions about more systems than you can reasonably test.
  • Suggestions:
    • Begin thinking about predictions after you have a hypothesis; however, if the hypothesis is proven false, the prediction will also fail.
    • Use a mathematical model to test your prediction.

Mathematical Models

  • Empirical vs. Deterministic
    • Empirical: Based on experimental observation.
    • Deterministic: Based on first principles (e.g., Density Functional Theory, Molecular Dynamics).
  • Keys to implementation:
    • All thesis experiments should have empirical models (at least).
    • If you do not have a mathematical model—only data—then you need to find an equation for least-squares fitting of your data.
    • If you encounter an interesting hypothesis that can be tested via deterministic models (first principles), consult with modeling to discuss the hypothesis and modeling.

Understanding of Methodology and Methods

  • Methodology is defined as:
    • "the analysis of the principles of methods, rules, and postulates employed by a discipline";
    • "the systematic study of methods that are, can be, or have been applied within a discipline"; or
    • "a particular procedure or set of procedures".
  • It can be viewed as: a collection of theories, concepts or ideas; comparative study of different approaches; critique of the individual methods.
  • Key point: Methodology refers to more than a simple set of methods; it refers to the rationale and philosophical assumptions that underlie a particular study.

Method

  • Method is a (systematic?) codified series of steps taken to complete a task or reach a specific objective.
  • The scientific method attempts to minimize the influence of the researchers' bias on the outcome of an experiment.
  • The researcher may have a preference for one outcome or another; it is important that this preference not bias the results or their interpretation.
  • Cautions:
    • Common sense and logic can tempt us to think no test is needed.
    • It is a mistake to ignore or rule out data that do not support the hypothesis.

No universal formal "scientific method"

  • There is no single universal formal method; there are several variants, and researchers tailor the process to the problem and their working methods.
  • Methodological choice: Scientist vs Engineer
    • A scientist asks "why?" and researches the answer to the question.
    • An engineer asks "how" to solve it and "how" to implement that solution, or how to improve it if a solution exists.
  • Core idea: scientists build in order to learn; engineers learn in order to build.

Quantitative vs Qualitative Methods

  • Quantitative methods and scientific rigor:
    • Data are numeric, collected with standardized methods, replicable, and analyzable with sophisticated statistical techniques.
  • Qualitative methods and scientific rigor:
    • Many qualitative approaches contend that there is no objective social reality; knowledge is constructed by observers influenced by traditions, beliefs, and social/political environments.

Four issues affecting the choice of method

  • 1. Credibility of findings
  • 2. Staff skills
  • 3. Costs
  • 4. Time constraints

Applications of Methods

  • Quantitative Methods
  • Qualitative Methods
  • Mixed Methods

Sample Size and Statistics

  • Defining the sample size from population:
    • n = rac{t^2 p q}{d^2}
  • Where:
    • n = the desired sample size (when population is greater than 10,000)
    • t = the standard normal deviate set at 2.58, which corresponds to the 99 percent confidence level (P < 0.01)
    • p = the proportion in the target population estimated to have a particular characteristic (set at 0.5)
    • q = 1.0 - p
    • d = Degree of accuracy desired, set at 0.05
  • Calculation example:
    • n = rac{(2.58)^2 imes (0.50) imes (0.50)}{(0.05)^2} = 665

Class Activity

  • Group activity: Using the four steps in the scientific method, design an outline to show how the scientific method can be applied to answer scientific questions.
  • Email group responses to the instructor: johnk@stjohns.edu