Quality and Credibility in Qualitative and Indigenous Research

Research Ethics

  • National Statement on Ethical Conduct in Human Research (2023)

    • Published by the Australian Government's National Health and Medical Research Council (NHMRC), the Australian Research Council, and Universities Australia.

  • Research Ethics – Values

    • Respect:
      • Each human has intrinsic value.
      • Respect autonomy.
    • Research Merit & Integrity:
      • Research involving human participants must have merit and the researchers must have integrity.
    • Justice:
      • Distributive justice.
      • Procedural justice.
    • Beneficence:
      • Research should do good.
      • Benefits of the research should justify the risks.

  • Research Ethics When Working with Indigenous Peoples

    • Respect
      • Respectful engagement with Indigenous peoples.
      • Value Aboriginal and Torres Strait Islander knowledge and wisdom.
    • Research merit & integrity
      • Research methods acknowledge cultural distinctiveness of Indigenous communities.
      • Identify potential negative consequences and design appropriate processes to minimise them.
    • Justice
      • Indigenous people should be treated as equal research partners.
      • Provide fair opportunity to Indigenous people to be involved with research.
    • Beneficence
      • Research benefits should advance interests of Indigenous peoples.
      • Discuss and agree on benefits with Indigenous research partners.

Indigenous Research Ethics – Core Principles

  • Sourced from the AIATSIS (Australian Institute of Aboriginal and Torres Strait Islander Studies) Code of Ethics for Aboriginal and Torres Strait Islander Research.
  • Responsibilities include:
    • Recognition and respect.
    • Engagement and collaboration.
    • Informed consent.
  • Integrity includes:
    • Cultural capability and learning.
    • Indigenous self-determination.
    • Indigenous leadership.
    • Indigenous lands and waters.
    • Ongoing Indigenous governance.
    • Sustainability and accountability.
    • Reporting and compliance.
    • Impact and value.
  • Responsibilities include:
    • Indigenous led research.
    • Indigenous perspectives and participation.
    • Indigenous knowledge and data.
    • Benefit and reciprocity.
    • Impact and risk.

Quality and Credibility

  • A Challenge: Avoid shallow, overly empathic discussions that lack analytical depth.
  • Comparison of Quantitative and Qualitative Research:
    • Quantitative:
      • Paradigm: Positivism/Realism.
      • Design: Fixed to control for variables.
      • Data: Experimental design, structured.
      • Analysis: Uses frequencies and statistics.
      • Sample: Seeks to generalise.
    • Qualitative:
      • Paradigm: Constructionism.
      • Design: Flexible and responds to context.
      • Data: Unstructured or semi-structured.
      • Analysis: Non-numerical analysis of text.
      • Sample: Depth of understanding rather than generalisability.
  • Quality = transparency of the research process; alternative name: Reliability.
  • Credibility = validation of analysis; alternative name: Validity.
  • Different philosophical stance → same or different evaluation criteria?
  • ‘Reliability’ and ‘validity’ do not involve the same procedures in qualitative and quantitative approaches.
  • Different approaches to quality and credibility across different qualitative research methods (Seale et al., 2004).

Study Design & Methods

  • Should take into account:
    • The phenomenon of interest.
    • Existing theories about this phenomenon.
    • The social and cultural context of this phenomenon.
    • The range of methods that could possibly be used to study this phenomenon.
  • Clear and explicit account of data generation methods.
  • Triangulation.
  • Triangulation Cautionary Note:
    • One should not adopt a naively 'optimistic' view that the aggregation of data from different sources will unproblematically add up to produce a more complete picture (Hammersley & Atkinson, 1983: 199).

Sample Size

  • “… one is also a number, the single case is also a quantity, and statistical significance is but one form of significance” (Schegloff, 1993: 101)
  • Generalisability may/may not be an objective.
  • Example: Studies using single-case analysis to explore complex phenomena. Jack Whalen's work on 9-1-1 calls and hysteria is cited.

Sampling

  • Representative Sample: Large amount of data; generally not suitable for rigorous qualitative research.
  • Purposive Sampling: An attempt to capture the diversity that you anticipate might be important
    • “seek out groups, settings and individuals where … the processes being studied are most likely to occur” (Denzin & Lincoln, 1994: 202).
  • Theoretical Sampling:
    • “When conducting theoretical sampling, you are much more selective than before about whom you obtain data from and what you seek from those individuals. You may focus on certain experiences, events or issues, not on individuals per se, because you want to develop your theoretical categories and need to define how and when they vary” (Charmaz, 1996: 46).
  • Convenience sampling.
  • Snowball sampling.

Analysis

  • Not all explanations are of equal value.
  • Clear and explicit account of steps and principles of analysis.
  • Comprehensive data analysis.
  • Avoiding anecdotalism.
  • Constant comparative method.
  • Deviant/negative case analysis.
  • Reporting (a representation) of data (e.g., transcription).
  • Providing ‘thick’ descriptions.
  • Demonstrating saturation.
  • Developing analysis with others.
    • Peer debriefing/data sessions.
    • Inter-rater agreement.
    • Respondent or communicative validation (‘member checking’).
    • “There is no reason to assume that members have privileged status as commentators on their actions” (Fielding & Fielding, 1986: 43).
  • Analysis Example:
    • Close, detailed and comprehensive analysis indicates a surprise is an interactional achievement (Wilkinson & Kitzinger, 2006).

Reflexivity

  • “Reflexivity is a form of critical thinking which aims to articulate the contexts that shape the processes of doing research and subsequently the knowledge produced” (Lazard & McAvoy, 2020: 160).
  • Underpinned by the broad question: “What is the research process and how am I influencing it?”
  • Pure objectivity and neutrality are not possible.
  • Personal reflexivity – what aspects of me influence the research process?
  • Epistemological reflexivity – what social, political, and historical contexts influence the research process?
  • Reflexivity Example:
    • Researchers acknowledge their own experiences with eating disorders to better understand interviewees' perspectives (Rance et al., 2017).

Being Reflective

  • Reflective journal.
    • Self-dialogue about researcher.
    • Private record of methodological decisions, preliminary reflections on data generation or analysis.
  • Multiple researchers/analysts.
    • Dialogue with others to develop complementary and deeper, or divergent and alternative perspectives.
  • Continuous questioning, multiple answers.
  • Being Reflective:
    • Stepping in – insider’s perspective: What do you already know? How does your current position, perspective allow you to engage with your topic and participants?
    • Stepping out – outsider’s perspective: How can you distance yourself from this position? Who can help you see it as an outsider?
    • Stepping sideways – alternative perspective: What other possible explanations are there? What other theoretical frames can you use?

Reporting Findings

  • Different structure - Less formulaic
  • More theoretical/conceptual
  • Often results and discussion presented together
  • Procedure of data generations and analysis need to be explained carefully
  • Data is usually presented:
    • According to themes, concepts, practices etc.
    • Active analysis/interpretation rather than mere description
    • Quotations in their context (social, interactional etc)
  • Enables readers to critically appraise the reported analysis

Critical Appraisal Tools

  • CONSOLIDATED CRITERIA FOR REPORTING QUALITATIVE STUDIES (COREQ) (Tong et al., 2007)
  • ABORIGINAL AND TORRES STRAIT ISLANDER QUALITY APPRAISAL TOOL (Harfield et al., 2020)

Summary

  • There is no ‘one size fits all’ approach to ensuring quality and credibility.
  • Differences to suit particular qualitative and Indigenous methods.
  • Differences according to study aims (e.g., whether generalization is an objective).
  • Caution should be exercised if using critical appraisal checklist.
  • Reflexivity is important in most qualitative research.
  • There are unique ways of reporting qualitative research.