Research Methods Notes

Social Science and Research

• Social science is important for understanding our identity, self-understanding, interpersonal relations, and society.
• It studies human beings in different types of societies and encompasses all social science disciplines.

Peter M. Blau's Study

• Peter M. Blau conducted an important study on the dynamics of bureaucracies.
• He examined their processes and changes using a case study of two American agencies.
• He employed participant observation, working at the agencies while collecting data, and used a questionnaire.
• He used both quantitative (recorded data as numbers) and qualitative (recorded data in text as field notes) methods.
• He used functionalism as a lens, viewing organizations as adapting to each other and society in ways that function best for both.

Reactivity and Reflexivity

• Reactivity/Control Effect: Knowledge may be biased or unreliable due to participants’ reactions to being involved in a research process.
• Reflexivity: Framing due to the researcher’s background and social experiences alters perceptions.

Positivism vs. Frankfurt School

• Positivism: Social scientific knowledge is developed through systematic studies of empirical facts about existing phenomena in society.
  • Claims that social science can relate to facts in society in the same way natural science does to facts in nature, without inherent differences.
• Frankfurt School (1960s): Emphasis on critical theory.
  • Argues that phenomena in society are not objective or observable facts.
  • There are no universal laws, as social conditions may change.
  • Focuses on involvement and interpretation.

Quantitative vs. Qualitative Data

• The book highlights the importance of the interpretation of data and position to society (observed from the outside or within), not the type of data.
• Metric Data: Most precise quantitative data (e.g., years).
• Non-Metrical Data: Expressed in numbers or categorical data (e.g., education categories: high or low).
• Data Expressed in Quantity Terms: (e.g., long, strong, few, etc.).
• Data Expressed in Text: (e.g., statements, opinions, etc.).
• The quantitative-qualitative divide is between 1-2 or 3-4.

Mixed Methods and Empirical Studies

• Mixed Methods Research: Some phenomena have both quantitative and qualitative aspects and are studied through both types of data.
• Empirical Studies: Questions about facts, including facts on values (e.g., how many people agree with this statement?).
• Normative Studies: Studies specific values, making claims about how conditions should be.

Postmodernism and Social Facts

• Postmodernist/Poststructuralists: Social reality is diverse, and there are no patterns to define as facts. There are multiple realities.
• The book argues that it is still possible to develop rational approaches.
• Émile Durkheim: Social Facts: Observable phenomena in society.
  • These are treated and analyzed as things, thus can be objective.
  • Treat them as data.
  • Social science should contribute to the improvement of society.
• Max Weber: Science should be based on facts but may be affected by values.
  • This is okay if scientists are aware of their biases.

Principles on Facts and Values

• Ontology: Social science is based on truth as a primary value.
  • Challenges established knowledge by seeking truth.
• Epistemology: Perceptions of truth in social sciences are theoretically, methodologically, and contextually founded.
  • Focus lies on how knowledge is developed; perception of reality is socially constructed.
• Methodology: Evaluations of truth in social science are based on rational and logical criteria.
  • Creates a common understanding of how to determine something is true.
  • Rationality and logic as the basis of methods.
• Paradigm Shift: One single study may create a completely new understanding, making previous approaches/results unimportant.

Formal Characteristics of Research Questions

1. Generality: Number of units. Is it Specific/ideographic or General/nomothetic?
2. Complexity: Number of features or relationships between features. Is it Simple/univariate, More complex/bivariate, or Most complex/multivariate?
3. Precision: Degree of accuracy in definitions of units.

Types of Research Questions

• Descriptive Questions: Ask how phenomena/conditions in society really are. Also called exploratory.
• Explanatory Questions: Why certain phenomena or conditions are the way they are. Focus lies on relationships between concepts.
• Interpretive Questions: How social phenomena can be understood in a larger/complex system.

Phenomena Dimensions

1. Time: Longitudinal studies. Focus lies on change, stability, and processes.
2. Space: Similarities and differences between multiple social contexts. Is comparative in nature.
3. Levels: Micro, meso, macro.
   • Contextual Studies: Multi-level research like macro-conditions’ impact on micro conditions.
4. Relations: Relational research, social networks, social structures.

Models

• Conceptual/Theoretical Model: Includes relationships between specific phenomena.
• Formal/Mathematical Model: Expressed in mathematical form in deductive system.
• Regression Analysis: Formal, quantitative model.

Steps in Formulating a Research Question

1. Identify the question.
2. Justify the question.
3. Make it more precise.
4. Operationalize.
   • Quan: Definitions in terms of degrees/quantities.
   • Qual: Definitions in terms of types/qualities.

Lecture Slides Highlights

• Most common types of RQs: exploratory, descriptive, explanatory.
• Needs to be focused, broadened for theoretical depth, neutral, and open-ended.
• Research Cycle: Research problem and literature review => research design => collecting data => analyzing data => report findings => repeat
• Deduction (mostly quan): theory => hypotheses => observation => confirmation
• Induction (mostly qual): observation => pattern => tentative hypotheses => theory
• Case-Based Research: Explore, describe, and explain a phenomenon and provide detailed, context-rich insights. Useful for theory-building.
• Variable-Based Research (Large-n): Describe and explain relationships between variables (IVs and DVs) by identifying trends. Useful for theory-testing.

Source-Critical Assessments

1. Availability: Limited/biased data.
2. Relevance: Contribution of relevant data.
3. Authenticity.
4. Credibility.

Units of Analysis and Observation

• Units of Analysis: Social elements that are emphasized in the RQ (e.g., families).
• Units of Observation: Key elements while collecting data (e.g., family members).

Types and Levels of Units

• Types of Units: Actors (individual or group), actions (single actors but also their interaction), opinions (statements), and events (whether they affect the actors).
• Levels of Analysis: Micro, meso, macro.
  • Multi-Level Analysis: Units at different levels in the same study.
• Wrong-Level Fallacy/Ecological Fallacy: Drawing false conclusions about members of an organization after studying the organization as a unit.

Types of Information About Relationships

1. Formal Properties: Strong/weak ties, (in)formal, etc.
2. Substantial Aspects: What does the relation concern, how are the units connected?

Temporal Studies

• How large/small social processes progress.
• Longitudinal Studies: Analyses of modes of development. Biographical studies recall data.
• Qualitative Content Analysis: Documents with no comparable content at different times.
• Time Series Data: Repeated questions at regular times. Combine data to express a trend.
• Panel Data: At random times. Does have the problem of drop-outs.
  • Panel data sees respondents as individuals; changes in their answers are considered gross changes, more detailed analysis.
  • Time series data sees them as a whole; changes are net changes.
• Cohort Analysis: Analyses on basis of people's age. Cohort has experienced a significant event at the same time.
  • Difference between cohorts is called cohort effect/generation effect.
  • Within a cohort => age effect/life-phase effect.
• Temporal Fallacies: Drawing conclusions about development while studying one point in time.
• Synchronous Data: One point in time/case study.
• Diachronic Data: Multiple points in time/longitudinal studies.

Special and Comparative Studies

• Special Studies: Similarities or differences between places, both geographically and about conditions in context. Societies.
• Comparative Studies: Comparing different societies or conditions in different societies. At least 2 units are systematically compared to find a causal relation.
  • Units as different as possible: find 1 commonality.
  • Units as similar as possible: find 1 difference

Equivalence in Comparative Research

• In order to compare particular phenomena, we must have equivalent data about these phenomena.

1. Linguistic: Have the same expressions and meaning across the compared societies.
2. Contextual: Different contexts, does the phenomenon have the same relevance?
3. Conceptual: Do concepts have the same meaning (culture bound)?
4. Methodological: Do the same methods create the same kind of data?

Variables in Multi-Level Analysis

1. Global Variables: They only refer to one level of the analysis.
2. Aggregated Variables: Variables at one level are used in the analysis as expressions of units at a higher level.
3. Contextual Variables: Based on one level and used in the analysis of units at a lower level.

Cross-Level Fallacies

• Conclusions about conditions at one level based on data from another level.
• Aggregative Fallacy: Faulty conclusions based on data about a higher level.
• Atomistic Fallacy: Faulty conclusions based on data about a lower level.

Historical-Comparative Studies

• Combines analyses of stability, change, and different levels in different societies.

Lecture Slides on Context and Units

• Context: The broader context/field/topic of the study.
• Unit of Analysis: Social units or elements that are the focus of the study (what is the study about?).
• Unit of Observation: Unit actually being observed in a study.
• Concept: An abstract/category that enables researchers to clarify, categorize, and understand a phenomenon in the social world.
• Variable: A characteristic that can be measured, is numeric (measurable), and they can vary between observations. Created through the operationalization of concepts.

Case Study Research

• Engages in an empirical inquiry that investigates a particular phenomenon in real-life within a specific bounded system.
• Treats cases as holistic and complex units.
• Defined as the intensive analysis of a single unit, where the researcher’s goal is to understand a larger class or similar units.

Types of Cases

1. Typical Case: Being representative of a larger population, what is average.
2. Extreme Case: Extreme version of the larger pattern/outlier. Chose a case that’s as far away from average as possible. Aim is to explain why the case is extreme.
3. Deviant Case: A case that does not fit the larger pattern. Aim is to explain why something is (not) happening, possibility for a new theory.

Comparative Case Studies

• Compare either same point in time and place but different case, or different points in time, or same case but across time. Important is comparability: unit homogeneity/equivalence.

Types of Comparative Case Studies

1. Diverse Cases: Two or more cases representing variation on a relevant condition. Cases are selected to represent the full range of values on a relevant condition/relationship.
2. Most Similar Systems Design: The cases are similar in nearly all areas but differ in one - why?
3. Most Different Systems Design: The cases differ in basically everything but have a similarity - why?

Large-N Studies and Survey Questions

• Planning survey questions: research question => theoretical concepts of interest => operationalization => variables => data collection and analysis
• Operationalization: Criteria for how concepts are going to be measured by empirical data.
• Independent Variable: A variable in the analysis of the relationship that assumes to influence another variable.
• Dependent Variable: A variable in the analysis of the relationship which is assumed to be influenced by one or more variables.
• Hypotheses: Statement about social phenomenon that can be tested empirically.
• Null Hypothesis: There is no significant relationship => reject.
• Alternative Hypotheses: There is a significant relationship between the variables => accept.

Types of Sources

1. Actors: Observed during action.
2. Respondents: Give answers to researchers’ questions, or informants, usually questioned about other actors.
3. Documents: Written, oral, or visual presentations, studied in content analyses.

Data

• Information that has been processed, systematized, and recorded in a specific form and for the purpose of a specific analysis.

Major Types of Research Design

1. Ethnographic Research/Participant Observation: Researcher is a participant in the process that is studied.
2. Structured Observation: No participation; observations are registered on a prepared schedule.
3. Unstructured Interviews: Conversations, not pre-determined questions; also, semi-structured interviews.
4. Questionnaires/Surveys: Fixed questions, fixed options. Survey experiment: use of experiment group and control group.
5. Qualitative Content Analysis: Discourse/narrative/opinions analysis (e.g., newspaper study).
6. Quantitative Content Analysis: Structured coding system with categories (e.g., tweet study).

Elements of Quan vs. Qual Research

Procedure for Selecting Information

1. Specify and define each of the concepts of the study.
2. Decompose concepts: specify dimensions for each term.
3. Define a set of categories for each dimension.
4. Clarify operational definitions.

Levels of Measurement for Variables

1. Nominal Level: Inequality between values (e.g., gender).
2. Ordinal Level: Rank order between values (e.g., education).
3. Interval Level: Distance between values (e.g., temperature/degrees).
4. Ratio Level: Proportion between values, with a meaningful or natural zero value (e.g., age).

Big Data Sources

1. Volunteered Information: Social media, transactions, sousveillance, crowdsourcing, etc.
2. Automated Information: Various systems like surveillance devices, scan data, interaction data.
3. Directed Information: CCTV, drones, individual identification.

Data Scraping and Mining

• Data Scraping: Using data programs to extract relevant info.
• Types: Web scraping, report mining, screens scraping.
• Data Mining: Also identifies patterns in the extracted data.

Sample Studies

• Part of the population is chosen to form a sample. Only part is studied, but findings are used to generalize.
• Statistical Generalization: Based on numerical data, extensive research.
• Theoretical Generalization: Based on conceptual relevance, intensive research.

Types of Samples

1. Population Sample: All units in the study’s universe.
2. Pragmatic Sample: Not meant to generalize, more exploratory/pilot studies.
3. Probability Sample: All units have a known probability of being included in the sample.
   • Confidence interval/statistical margin of error.
   • Significance level: P<0.05
4. Strategic Sample: Theoretical understanding of the social conditions being studied. To develop theories (analytical induction) or to make a holistic generalization.
5. Case Studies: Restricted to one unit.

Sampling Methods

1. Simple Random Sampling: Random drawing from a list of all units in the study’s universe.
2. Systematic Sampling: Sampling of every Nth unit on a list of all units in the universe (e.g., every 10th unit is used).
3. Stratified Sampling: Units are divided into categories according to their properties, random drawing of units from each category.
   • Proportional sample: All units have the same probability of being included in the sample.
   • Different per category: Disproportional. Weighing: Under-represented units receive more weight.
4. Cluster Sampling: Units divided into clusters according to location, random drawing of entire clusters.

Multi-Stage Probability Sampling

• Different sampling methods are used in turn.

Methods for Strategic Sampling

1. Quota Sampling: Units are divided into specific categories from which a specific number (quota) is selected.
2. Haphazard Sampling: Sampling of units that happen to be located in a particular place at a particular time.
3. Self-Selection: Consists of actors who volunteer to participate (often surveys).
4. Snowball Sampling: First actor suggests more actors to participate for the sample.

Lecture Slides on Quantitative Research Process

1. Surveys
2. Survey experiment: A researcher randomly assigns participants to at least two experimental conditions (vignettes). Aims to identify if there is a relationship between the manipulated variables.
3. Documents and records/archival data (large-n data sets)

Qualitative Research Process

1. Interviews (structured, semi-structured, unstructured)
2. Focus groups
3. Participant observation/ethnographic research: research directly observe actors in their natural environment. Use of extensive fieldnotes and either overt or covert participation observation.
4. Documents and records/archival data

Sampling Considerations

• Deciding your units of observation
  • Population Theoretical: Who/what do you want to study?
  • Study Population: Who/what do you have access to?
  • Sampling Strategy: How do you access the participants/data?

Types of Observations

• Cross-Sectional: Observations at one point in time.
• Panel: Observe the exact sample people over time.
• Cohort: Observe people who shared an experience over time.
• Trend: Observe different people over time.

Probability Sampling (Quan)

• Goal is being able to generalize. Aims to ensure that each unit of observation in the population has an equal chance of being included in the study.

1. Simple random sample
2. Systematic sample: every nth person is chosen
3. Stratified sample: population is divided into sub-groups and a random sample is selected from each sub-group
4. Clustered sample: divide population into clusters (groups) based on physical/geographical proximity, randomly choose one cluster to study

Sampling and Non-Sampling Error

• Sampling Error: Error that arises in a data collection process as a result of taking a sample from a population rather than using the whole population.
• Confidence Interval: Probability that sample accurately reflects the population (standard is 95%).
• Margin of Error: Range by which the population parameter may deviate from the sample parameter.

Non-Sampling Error

1. Population Specification Error: Does not understand who to collect data on.
2. Sample Frame Error (Selection Bias): Wrong sub-group is used for representation.
3. Self-Selection Error (Volunteer Basis): Only those who are interested respond.
4. Non-Response Error: Inability to contact potential respondents or their refusal.

Non-Probability Sampling (Qual)

• No aim of equal change of being included. Continue to sample until you don’t get new information.

1. Convenience sample: volunteers who are available
2. Quota sample: research determines what kind of characteristics are wanted in the sample. Minimum of two groups for comparison.
3. Purposive sample: researcher decides who is included. Specific individuals are targeted.
4. Snowball sample: when reaching difficult populations. Respondents help find new respondents.

Number of Variables

• Bivariate Analysis: Study with 2 variables.
• Multivariate Analysis: Study with 3 or more variables.
• Table Analysis: No fixed limit of variables in a table, but focus lies on 2 or 3.
• Correlation Analysis: Primarily bivariate. Correlation is clarified between each pair of variables. Expressed in correlation coefficient that are presented in a correlation matrix (built on separate bivariate analyses).
• Regression Analysis: Multivariate, suitable for analyzing the relationship between many variables. Variables at interval or ratio level, at the nominal level only if they are dichotomies with values 0 and 1 (dummy variables).

Dependency Relation Between Variables

• Independent Variable: Exerts an effect.
• Dependent Variable: Is affected.
• Symmetrical Relation: In the analysis, the variables are treated as equivalent.

Determining Dependency

1. Chronology: Can we assume that each unit’s value for one variable is determined before they got their value for the other variable (e.g., gender first, income later)?
2. Causal Relation: Independent variable is considered the cause, the dependent variable the effect.
3. Spurious: Bivariate relationship is due to a statistical relationship between one of the two variables and a third variable. That first relationship will disappear when analyzing the 3 variables together. (e.g., income – home size – age)

Impressionist Approach to Data Analysis

• Conclusions drawn from researcher’s experience and impressions.

Coding

• Finding keywords (codes) that describe a larger section of the text. In quan => codes receive a number or value before the analysis

Types:

1. Descriptive: Characteristics of actual or explicit text
2. Interpretive: Researcher’s interpretations
3. Explanatory: Researcher’s explanation of explicit elements of the text

• Open Coding: Initial characterization of the key elements of the data, mostly descriptive codes
• Categories/Type: Collection of specific common characteristics. Is more systematic than open coding
• Concept: Theoretical construct/general notion for a particular type of phenomena

Constant Comparative Methods

• Repeated systematic comparisons of the various elements in the data. Useful for theory development => grounded theory.
• Typology: Multiple categories are arranged in relation to each other in a particular system.
• Ideal Type: Representation of a particular phenomenon, where the most important features of the phenomenon are isolated and described in an idealized or pure form. Roughly a model, not ideal in the normative sense.
• Matrix: Chart for systematizing and arranging quotes from qualitative data. Can be person-centered or theme-centered. Filled with text rather than numbers => figures to illustrate structural patterns.
• Sociograms: Actors are presented as points, with relationships between the actors illustrated by lines or arrows between the points.
• Hierarchies: Concern hierarchical structures, where the relationships represent relations between superior and subordinate categories or units. In an organization: organizational chart.
• Condensation: Content of the data is presented in an abbreviated or condensed form, method to obtain a comprehensive understanding.
• Narrative Analysis: Text is organized with reference to typical elements in a story or narrative.
• Discourse Analysis: Used to establish a comprehensive understanding of expressions of opinion and communication processes
  • Discourse: System of ideas, perceptions, and concepts about conditions in society
• Grounded Theory/Analytical Induction/Theory-Generating Studies: Hypotheses and theories based on empirical evidence. Empirical patterns are not only described but also interpreted.
• Search for Deviating Cases: Search for elements in data that do not conform to the hypotheses to formulate a new, complete theory (only in qual => in quan, the hypotheses are rejected).
• CAQDAS: Computer-assisted qualitative analysis software. Often depends on coding, effective in the constant comparative method. Not effective to find a holistic understanding/finding overarching patterns/interpretation of data.

Examples: NVivo, ATLAS.ti, N6, MaxQDA

Univariate Distribution

• Distribution on a single variable or in a single index.
• Frequency: Number of units registered with a particular value
  • Absolute frequency: actual counts
  • Relative frequency: percentage of total counts, total always 100%
• A relative distribution/percentage distribution is a better basis for comparisons.
• Cumulative Frequencies: Share of a particular value, plus all the lower values. Highest is thus always 100%. Either absolute or relative.
• Inverse: starting from high to low.
• Graphical: Not as a table but as a graph or chart. Examples: bar chart, pie chart, line chart (line is called the curve).
• Symmetrical distribution is called the normal distribution or bell curve.

Statistical Measures - Central Tendency

1. Nominal: modes (variable with the highest frequency)
2. Ordinal: median (divided the units in 2 equal parts when arranged in ascending order)
3. Interval/ratio: either modes or median. Also uses mean (values of units/number of units).

Dispersion

1. Standardized modal percentage: 1 (max dispersion) to 100 (no dispersion)
2. Quartile deviation: units divided into 4 parts. Difference between Q1 and Q3. Quartile deviation is (Q3-Q1)/2
3. Variance or standard deviation: based on distance from a unit to the mean. Usually at the interval and ratio level.

Lecture Slides on Quantitative Analysis

Types:

1. Descriptive Statistics: Explaining the basic features of the data.
2. Inferential Statistics: Aims to draw conclusions (inferences) from the data that can be generalized to a larger population.

Measurement Levels

1. Nominal: Data with no inherent order or ranking (e.g., nationality, gender, religion).
2. Ordinal: Data with an inherent order or rank (e.g., age group, level of education).
3. Scale/Continuous: Ordered data with a meaningful metric (e.g., GDP).

Descriptive Statistics

Allow us to summarize and display information about a single variable, also called univariate analysis

1. Frequency/percentage distribution: describes how the sample is distributed
2. Measures of central tendency: mean (average), median (middle point), mode (most common/frequent)
3. Measures of variation: range (highest value - lowest value), interquartile range (Q3-Q1), standard deviation (average spread of the data around the mean. The larger the SD, the more spread out the data is

Outliers

• A data point that differs significantly from other observations. Can skew the results and SD.
• Needs to be 1.5 \, x \, IQR above Q3 or below Q1

Bivariate Analysis

• Two variables are analyzed to determine their relationship.
• Correlation Coefficient: Measured the degree of linearity in the relationship between the variables between -1 and +1.
• Weak (-0.2 to 0.2), medium (-0.2 to -0.4 or 0.2 to 0.4), or strong (
• Statistical Significance: Tells us if a relationship exists that is not based on chance. P-value is used to determine this. Is between 0 and 1, threshold p<0.05 is used.
• If p

Multivariate Analysis

• Allows the simultaneous investigation of the relationship between more than two variables. By adding more variables, we are able to control for differences in the sample.

Types of variables

1. Independent
2. Dependent
3. Control: anything that is held constant or limited in a research study. It is not of direct interest to the study’s objectives but may have some impact
4. Confounding variable: a variable that both influences the IV and DV, causing a spurious association

• Spurious Correlation: Two events are found to be correlated despite having no logical connection.

Inferential Statistics

• Types of analysis used to determine something about a population based on a sample. Used for hypotheses testing

Qualitative Analysis

Types:

1. Content Analysis: Subjective interpretation of text data through systematic coding, condensing the raw data into categories. Develop a codebook in which each code is clearly defined with in-/exclusion criteria.
2. (Reflexive) Thematic Analysis: Identifying, analyzing, and reporting patterns (themes) within data.
3. (Critical) Discourse Analysis: Examine patterns of language across texts and consider the relationship between language and the social and cultural contexts in which it is used. Language is not neutral.
4. Narrative Analysis: Discourses with a clear sequential order that connects events in a meaningful way. How do people make sense of what happened?

Data Quality

• Based on scientific principles of truth and logical discussion, selection of units and data collection are carried out in a systematic matter, in accordance with the used research design.

Criteria

1. Reliability: Accuracy and trustworthiness, often tested by repeating research (quan).
2. Validity: Adequacy and relevance; need consistency in theoretical and operational definitions. Reliability is needed to have validity.

Reliability Types

1. Stability: Consistency during repetition

• Tested by the test-retest method: repetition with a random sample of units

1. Equivalence: Comparison of data between research with the same design with different people doing the study

• Tested by the inter-subjectivity method: comparisons of different observers/interviewers. Deviations in findings are due to the observer’s reliability.

1. Equivalence between different indicators in the same index

• Tested with the split-half method/internal consistency test: data in a single study is compared on different parts of the research design

• Reliability is measured on a scale from 0 (no consistency) to 1 (full consistency.) Does not take random consistency into account.

• Scott’s pi: \Pi = (\% \, actual \, consistency - \% \, random \, consistency) / (100\% - \% \, random \, consistency)

• In qual => reliability = credibility: findings are based on data about actual, objective, conditions.

• Internal Consistency: Between different elements of the data within the study. Parts of the data fit well in the bigger picture.

• External Consistency: Between data in the study and other available information about the conditions studied. About the bigger picture.

Validity Types

1. Face Validity: Data collection is appropriate for the intention of the study
2. Quan Definitional:
   • Theoretical (intent on what to study)
   • Operational (what actually is being studied)
   • Content validity => operational definition is broad
   • Criterion validity => consistency between data of the same concepts, different definitions
   • Concurrent/predictive => two tests at the same time, of which one is already established, to test the newer one
   • Construct validity => data on the relationship between concepts aligns with the known relationship between then. Is either convergent (there is a relation) or discriminant (lack of a relation)
3. Internal Validity: Experiment is satisfactory
4. External Validity: Results are realistic/generalizable
5. Qual Competence Validity: Researcher’s competence to collect the data, experienced enough
6. Communicative Validity: Discussion between researcher and others about the data

• Directly with the sources: actor validation
• With colleagues: collegial validation, offers different academic perspectives

1. Pragmatic Validity: Good basis for action, prescriptions, and suggestions made by the study

Lecture Slides on Evaluating Quantitative Research: Rigor

• Measurement Error: The difference between the observed value and the actual value
• Random Error: Creates imprecision in data and impacts how reproducible the result would be. Reduce by repeating, increasing sample size, and controlling variables
• Systematic Error: Creates bias in data and means that measurements of the same thing will vary in predictable ways. Reduce by triangulation and calibration

Reliability

• Accuracy and consistency of data collection

1. Stability and test-retest
2. Equivalence: internal consistency. Assesses whether you get the same results when testing a specific concept via multiple items (like survey questions)
3. Equivalence: inter-subjective: refers to the degree in which multiple observers agree on a measurement. Is called inter-rater reliability

Validity

• Adequacy and relevance of data collection

1. Measurement Validity:

• Content: does the measure adequately capture the concept’s full meaning?
• Criterion (concurrent): measures the consistency between different measures for the same concept
• Construct: Shows the relationship between measures for different concepts. Strong correlation between concepts is convergent validity. Weak correlation is discriminant validity

1. Design Validity:

• Internal: does the IV cause the DV? Only tested in (survey) experiments
• External: population and ecological: can it be generalized to the population and across settings?

Qualitative Research: Trustworthiness

1. Credibility => establishing that the results are believable
2. Transferability => applicability of findings to other settings
3. Dependability => accounting for the changing context in which the research is conducted
4. Confirmability => can it be confirmed by other researchers
5. Reflectivity => open about their positionality

Basic Ethical Norms

1. Communalism: Open available research
2. Universalism: Pure academic criteria, objectivity
3. Dis