Quantitative Data Types, Measurement Validity, Scale Development, and Social Cohesion Network Analysis

Course Logistics and Administrative Announcements

• Class Context & Attendance:

  • Attendance is lower than usual due to students preparing for their upcoming assignment.

  • Completion of the course survey is compulsory for all enrolled students.

• Seminar Scheduling Changes:

  • Hayley is unwell, resulting in the cancellation of today's in-person seminar.

  • An online replacement seminar is scheduled for tomorrow (Tuesday) at $6\,\text{PM}$.

• Post-Lecture Consultation:

  • Following the lecture recording, a period of $10\text{--}15\,\text{minutes}$ is set aside for individual or group discussions regarding the assignment.

Foundations of Quantitative Data Analysis and Software Considerations

• Core Principle of Quantitative Research:

  • Quantitative research transforms all collected data into numeric values for analysis.

  • Not all numerical values represent the same mathematical properties or carry identical meaning.

• Impact of Data Types:

  • The structural type of data collected dictates both the formulation of research questions and the specific statistical analysis procedures that can be legitimately applied.

  • Applying inappropriate statistical techniques to certain data types yields invalid analytical results.

• Software Handling across Applications:

  • Dedicated Statistical Software (e.g., SPSS): Requires explicit manual classification of variables (e.g., specifying whether a variable is categorical) so the software automatically restricts forbidden mathematical operations.

  • Spreadsheet Applications (e.g., Excel): Does not automatically enforce data type restrictions. Researchers must manually maintain diligence to prevent treating categorical data as continuous variables.

Taxonomy of Data Types in Quantitative Research

• Categorical Data:

  • Definition: Data partitioned into distinct groups or categories that possess distinct qualitative properties.

  • Numerical Representation: Assigning numbers to categories serves purely as a labeling mechanism and does not confer quantitative magnitude or value.

  • Examples:

    • Gender: Female assigned as $1$, Male assigned as $2$, Non-binary assigned as $3$, alongside other prospective categories. Assigning Male as $2$ and Female as $1$ does not imply that Males count twice as much as Females.

    • Census Ancestry: Categories such as Australian, First Nations, Chinese, or British. Assigning Chinese as $1$ and British as $2$ does not quantify British as double the value of Chinese.

    • Hair Color: Discrete qualitative groupings such as blonde or dark.

  • Mathematical Restrictions: Calculating summary statistics such as the mathematical mean on categorical variables is invalid (e.g., calculating the mean of a gender column produces a meaningless number).

• Ordinal Data:

  • Definition: Data points that preserve a clear sequential or rank order (e.g., from lowest to highest), but where the precise distance or interval between adjacent scale points is non-uniform and unknown.

  • Likert Scales:

    • Widely used in survey research (and present in Assignment $3$ data files).

    • Standard 5-point coding layout:

      • $1$ = Strongly disagree

      • $2$ = Disagree

      • $3$ = Not sure / Midpoint

      • $4$ = Agree

      • $5$ = Strongly agree

    • Scale Properties: The numeric labels denote sequence ($1$ precedes $2$), but the subjective distance between $1$ (Strongly disagree) and $2$ (Disagree) may not equal the distance between $2$ (Disagree) and $3$ (Not sure), as respondents may perceive major psychological jumps only at the extreme ends of the scale ($1$ and $5$).

  • Educational Level Example:

    • Primary school = $1$

    • Secondary school = $2$

    • Diploma = $3$

    • Bachelor degree = $4$

    • Scale Properties: Represents a clear progressive order, but the educational gap between primary and secondary school does not mathematically mirror the gap between a diploma and a bachelor degree.

• Numerical Data:

  • Definition: Values representing genuine mathematical numbers with consistent, fixed intervals between units.

  • Statistical Applicability: Permits all standard mathematical calculations, including mean, standard deviation, and advanced inferential statistics.

  • Subtypes:

    • Discrete Numerical Data: Countable integer values where fractional components are non-meaningful. Example: Number of children in a household ($1$, $2$, or $3$ children; values like $1.2$ or $1.5$ children are impossible).

    • Continuous / Amount Numerical Data: Standard continuous quantities and financial measures where fractional values remain meaningful. Examples: Weekly income (e.g., $\$1,200$ or $\$1,500$) and age.

Measurement of Latent Social Constructs

• Latent Social Constructs:

  • Definition: Abstract social phenomena that cannot be directly observed or measured in a physical manner.

  • Characteristics: Unobservable directly, non-tangible, and subject to contested conceptual definitions.

  • Examples: Hate, misinformation, polarization, extremism, prejudice, antisemitism, homophobia, and social cohesion.

• Operationalization via Observable Indicators:

  • Direct inquiry is frequently ineffective due to social desirability bias (e.g., asking individuals direct questions such as "How much do you hate this group?" yields untruthful responses).

  • Indicator Approach 1: Machine Learning & Digital Media Speech:

    • Artificial intelligence (AI) classifiers are built using machine learning techniques and trained on explicit definitions to detect hateful language in digital text.

    • The trained AI assigns a probability score ranging from $0$ to $100$ to individual pieces of content to measure hate at scale.

  • Indicator Approach 2: Community & Official Incident Reports:

    • Tracking external documentation submitted by targeted communities to formal bodies, such as the Human Rights Commission or community reporting tools like the Islamophobia Regista.

    • Systematic analysis of logged reports (e.g., verbal abuse incidents on public transport) serves as an empirical proxy for societal hate levels.

Frameworks of Measurement Validity and Reliability

• Measurement Validity:

  • Core Concept: The degree to which an instrument, survey question, or testing procedure accurately measures the exact construct it claims to measure.

  • Content Validity: The extent to which a measurement tool comprehensively represents all theoretical dimensions and facets of a social construct (e.g., capturing all $10$ identified dimensions of antisemitism or $5$ types of Islamophobia derived from systematic literature reviews).

  • Criterion Validity: The extent to which operational measurements correlate with external criteria or specific outcomes.

    • Predictive Validity: Degree to which measurement scores correlate with real-world longitudinal outcomes (e.g., comparing a daily timeline of online hate speech probabilities generated by AI classifiers against official offline incident timelines presented to bodies such as the Antisemitism and Royal Commission).

    • Concurrent Validity: Degree to which a new measurement tool co-varies predictably with existing, theoretically validated measures evaluated at the same time (e.g., validating a new hate scale by demonstrating that participants scoring high on hate also score high on established prejudice scales).

  • Construct Validity: The precision with which a tool isolates and measures the target construct rather than adjacent or confounding concepts (e.g., ensuring an antisemitism tool specifically measures anti-Jewish hate rather than criticism of Israel, anti-Muslim hate, or broader anti-immigration sentiment).

  • External Validity: The generalizability of a measurement instrument and its findings across different population settings, samples, and external contexts.

  • Consequential Validity: An evaluation of the ethical, political, policy, and practical societal ramifications that stem directly from deploying a specific measurement tool in real-world environments.

• Measurement Reliability:

  • Core Concept: The consistency, stability, and repeatability of a measurement instrument across repeated deployments and contexts.

  • Test-Retest Reliability: The stability of measurement scores obtained from the same test over time.

  • Inter-Rater Reliability: The degree of consensus among multiple independent human annotators coding identical data according to a shared definition (used to evaluate whether observational discrepancies stem from ambiguous construct definitions, flawed instruments, or annotator error).

  • Internal Consistency: The degree to which separate sub-dimensions or items of a unified conceptual scale move together systematically across measurements.

• Target Visualization Metaphor:

  • Poor Validity & Poor Reliability: Shots scattered across the target board without forming a cluster and missing the bullseye center.

  • Good Reliability & Poor Validity: Shots tightly clustered together in a off-center location, consistently missing the bullseye center.

  • Good Reliability & Good Validity: Shots tightly clustered directly within the bullseye center.

Development and Validation Protocol for Social Measurement Scales

• Case Study: Development of a 12-Item Hate Scale:

  • Development Investment: Required $2\,\text{years}$ of work by a $5\text{-person}$ interdisciplinary research team (published in $2024$ or $2025$).

  • Stage 1 (Theoretical Definition): Grounded the operational framework in established literature on outgroup prejudice, adopting baseline definitions from social psychologist Old Port (from the $1960\text{s}$).

  • Stage 2 (Item Generation): Reviewed existing related measurement scales to extract an initial pool of approximately $90$ items capturing related theoretical constructs.

  • Stage 3 (Expert Panel Review): Conducted multi-day review workshops with subject-matter experts to evaluate item phrasing, remove flaws, and draft additional items covering missing dimensions of hate, expanding the pool to approximately $100$ items.

  • Stage 4 (Pilot Testing): Administered the $100\text{-item}$ pilot instrument to a initial data collection sample of $N = 500$ participants.

  • Stage 5 (Statistical Reduction): Conducted factor-analytic statistical tests to identify redundant items measuring identical latent variance, eliminating duplicate items while retaining items that captured distinct sub-dimensions.

  • Stage 6 (Structural Finalization): Established a finalized scale of $12$ items organized across $3$ distinct, theoretically coherent sub-dimensions.

  • Stage 7 (Cross-Sample Validation): Administered the finalized $12\text{-item}$ scale to $2$ separate, independent replication samples of $N = 1,200$ participants in the United States using different panel providers to verify that the scale maintained uniform statistical properties across diverse cohorts.

• Applied Recommendation for Survey Design:

  • Researchers, social workers, and psychologists should avoid drafting unvalidated, ad-hoc survey questions.

  • Practitioners should utilize established, pre-validated scales to ensure structural validity and prevent measurement error.

Operationalizing Social Cohesion Through Multi-Layer Network Analysis

• Research Context & Policy Background:

  • Project funded by the New South Wales government and Department of Home Affairs to evaluate societal cohesion.

  • Public inquiries (e.g., antisemitism and social cohesion inquiry) frequently debate social cohesion, but operational measurement remains difficult.

• Limitations of Traditional Measures:

  • Traditional survey research is expensive and relies on indirect proxy variables (e.g., the Scanlon Foundation operationalizes cohesion through measures of trust, intergroup conflict, and prejudice rather than direct measures of social cohesion).

• Framework Publication & Development Phase:

  • Required $6\,\text{months}$ of preliminary theoretical conceptualization resulting in two foundational research papers:

    • Paper 17: Reviewed global and Australian measurement frameworks, evaluating their analytical strengths, weaknesses, and contextual suitability.

    • Paper 18: Established a novel operational definition tailored to measuring social cohesion in digital network data.

• Network Analysis Operational Framework:

  • Interaction Data: Evaluates digital connection nodes (retweets, shares, comments, and direct interactions) across millions of network data points.

  • Baseline Issue Polarization: On controversial subjects (e.g., immigration policies), networks naturally split into opposing camps characterized by high internal connectivity and inter-camp hostilties/toxic interactions (identified using machine learning toxic language classifiers).

  • Multi-Layer Cross-Topic Tracking: Measures whether the polarization observed on contentious primary topics persists when the exact same individuals discuss unrelated secondary domains (e.g., sports, vaccines, health, exercise, movies, or general entertainment).

  • Structural Conditions for Social Cohesion:

    • Low Social Cohesion: Societal fracture lines are rigid and identical across all domains (i.e., individuals who are opposed on immigration completely refuse to interact or share common space when discussing sports, health, or entertainment).

    • High Social Cohesion: Societal fracture lines are fluid across domains (i.e., individuals strongly disagree on contentious political issues, but form shared positive connections or neutral interactions within secondary layers such as sports fandom, health discussions, workplace environments, or social venues).

Questions and Discussion

• Toxicity and Disagreement Thresholds:

  • Audience Query: Raised the issue of how toxicity levels relate to social cohesion, questioning where the structural boundary lies when individuals use toxic language or disagree strongly on political matters, yet maintain personal or professional relationships (e.g., co-workers or socializing at a pub).

  • Response: A cohesive society does not require universal consensus; functioning democratic societies naturally accommodate widespread disagreement, provided individuals disagree without expressing mutual hatred. Disagreement and passionate language do not automatically destroy social cohesion unless political divides dictate and sever every other layer of social interaction.