Qualitative Research in Healthcare: Assessing Quality

Qualitative Research in Health Care: Assessing Quality

Introduction

Qualitative methods have become more common in health services research and health technology assessment.
Increased reporting of qualitative research studies in medical and related journals.
Interest in these methods has led to scrutiny of qualitative research.
Researchers want to understand qualitative methods and claims made from findings.
The quality of methods used determines the status of all research.
Concern about assessing quality has led to guidelines for qualitative work.
Users and funders of research play a role in developing these guidelines.
They require means of assessing quality and distinguishing "good" and "poor" quality research.
The issue of "quality" in qualitative research is part of a larger debate about the nature of knowledge produced, whether its quality can be judged, and how.
This paper outlines two views of how qualitative methods might be judged.
Qualitative research can be assessed according to validity and relevance.

Two Opposing Views

Debate over whether qualitative and quantitative methods can and should be assessed according to the same quality criteria.
Extreme relativists believe all research perspectives are unique and equally valid.
This position means research cannot derive unequivocal insights relevant to action and would command little support among applied health researchers.
Two broad, competing positions exist: for and against using the same criteria.
Within each position, there is a range of views.

Separate and Different: The Antirealist Position

Advocates argue that qualitative research represents a distinctive paradigm.
Should not be judged by conventional measures of validity, generalizability, and reliability.
Rejects naive realism—belief in a single, unequivocal social reality independent of the researcher.
Instead, there are multiple perspectives of the world constructed in the research process.
Relativists believe assessment criteria are feasible but distinctive ones are required to evaluate qualitative research and have put forward a range of different assessment schemes.
The choice and importance of quality criteria depend on the topic and purpose of the research.
Hammersley attempted to pull together these quality criteria.
These criteria are open to challenge (for example, it is arguable whether all research should be concerned to develop theory).
Many of the criteria listed are not exclusive to qualitative research.

Relativist criteria for quality

Degree to which substantive and formal theory is produced and the degree of development of such theory
Novelty of the claims made from the theory
Consistency of the theoretical claims with the empirical data collected
Credibility of the account to those studied and to readers
Extent to which the description of the culture of the setting provides a basis for competent performance in the culture studied
Extent to which the findings are transferable to other settings
Reflexivity of the account—that is, the degree to which the effects of the research strategies on the findings are assessed or the amount of information about the research process that is provided to readers

Using criteria from quantitative research: subtle realism

Authors agree that all research involves subjective perception and that different methods produce different perspectives.
Unlike antirealists, they argue that there is an underlying reality that can be studied.
The philosophy of qualitative and quantitative researchers should be one of “subtle realism”—an attempt to represent that reality rather than to attain “the truth.”
From this position, it is possible to assess the different perspectives offered by different research processes against each other and against criteria of quality common to both qualitative and quantitative research, particularly those of validity and relevance.
However, the means of assessment may be modified to take account of the distinctive goals of qualitative research.

Assessing the validity of qualitative research

There are no mechanical or “easy” solutions to limit the likelihood that there will be errors in qualitative research.
However, there are various ways of improving validity, each of which requires the exercise of judgment on the part of researcher and reader.

Triangulation

Compares the results from either two or more different methods of data collection (for example, interviews and observation) or, more simply, two or more data sources (for example, interviews with members of different interest groups).
The researcher looks for patterns of convergence to develop or corroborate an overall interpretation.
This is controversial as a genuine test of validity because it assumes that any weaknesses in one method will be compensated by strengths in another, and that it is always possible to adjudicate between different accounts (say, from interviews with clinicians and patients).
Triangulation may therefore be better seen as a way of ensuring comprehensiveness and encouraging a more reflexive analysis of the data than as a pure test of validity.

Respondent validation

Respondent validation, or “member checking,” includes techniques in which the investigator’s account is compared with those of the research subjects to establish the level of correspondence between the two sets.
Study participants’ reactions to the analyses are then incorporated into the study findings.
Although some researchers view this as the strongest available check on the credibility of a research project, it has its limitations.
For example, the account produced by the researcher is designed for a wide audience and will, inevitably, be different from the account of an individual informant simply because of their different roles in the research process.
As a result, it is better to think of respondent validation as part of a process of error reduction which also generates further original data, which in turn requires interpretation.

Clear exposition of methods of data collection and analysis

Since the methods used in research unavoidably influence the objects of inquiry (and qualitative researchers are particularly aware of this), a clear account of the process of data collection and analysis is important.
By the end of the study, it should be possible to provide a clear account of how early, simpler systems of classification evolved into more sophisticated coding structures and thence into clearly defined concepts and explanations for the data collected.
Although it adds to the length of research reports, the written account should include sufficient data to allow the reader to judge whether the interpretation proffered is adequately supported by the data.

Reflexivity

Reflexivity means sensitivity to the ways in which the researcher and the research process have shaped the collected data, including the role of prior assumptions and experience, which can influence even the most avowedly inductive inquiries.
Personal and intellectual biases need to be made plain at the outset of any research reports to enhance the credibility of the findings.
The effects of personal characteristics such as age, sex, social class, and professional status (doctor, nurse, physiotherapist, sociologist, etc) on the data collected and on the “distance” between the researcher and those researched also needs to be discussed.

Attention to negative cases

As well as exploration of alternative explanations for the data collected, a long established tactic for improving the quality of explanations in qualitative research is to search for, and discuss, elements in the data that contradict, or seem to contradict, the emerging explanation of the phenomena under study.
Such “deviant case analysis” helps refine the analysis until it can explain all or the vast majority of the cases under scrutiny.

Fair dealing

The final technique is to ensure that the research design explicitly incorporates a wide range of different perspectives so that the viewpoint of one group is never presented as if it represents the sole truth about any situation.

Relevance

Research can be relevant when it either adds to knowledge or increases the confidence with which existing knowledge is regarded.
Another important dimension of relevance is the extent to which findings can be generalised beyond the setting in which they were generated.
One way of achieving this is to ensure that the research report is sufficiently detailed for the reader to be able to judge whether or not the findings apply in similar settings.
Another tactic is to use probability sampling (to ensure that the range of settings chosen is representative of a wider population, for example by using a stratified sample).
Probability sampling is often ignored by qualitative researchers, but it can have its place.
Alternatively, and more commonly, theoretical sampling ensures that an initial sample is drawn to include as many as possible of the factors that might affect variability of behaviour, and then this is extended, as required, in the light of early findings and emergent theory.
The full sample, therefore, attempts to include the full range of settings relevant to the conceptualisation of the subject.

Is there any place for quality guidelines?

Whether quality criteria should be applied to qualitative research, which criteria are appropriate, and how they should be assessed is hotly debated.
It would be unwise to consider any single set of guidelines as definitive.
We list some questions to ask of any piece of qualitative research; the questions emphasise criteria of relevance and validity.
They could also be used by researchers at different times during the life of a particular research project to improve its quality.

Some questions about quality that might be asked of a qualitative study

Worth or relevance—Was this piece of work worth doing at all? Has it contributed usefully to knowledge?
Clarity of research question—If not at the outset of the study, by the end of the research process was the research question clear? Was the researcher able to set aside his or her research preconceptions?
Appropriateness of the design to the question—Would a different method have been more appropriate? For example, if a causal hypothesis was being tested, was a qualitative approach really appropriate?
Context—Is the context or setting adequately described so that the reader could relate the findings to other settings?
Sampling—Did the sample include the full range of possible cases or settings so that conceptual rather than statistical generalisations could be made (that is, more than convenience sampling)? If appropriate, were efforts made to obtain data that might contradict or modify the analysis by extending the sample (for example, to a different type of area)?
Data collection and analysis—Were the data collection and analysis procedures systematic? Was an “audit trail” provided such that someone else could repeat each stage, including the analysis? How well did the analysis succeed in incorporating all the observations? To what extent did the analysis develop concepts and categories capable of explaining key processes or respondents’ accounts or observations? Was it possible to follow the iteration between data and the explanations for the data (theory)? Did the researcher search for disconfirming cases?
Reflexivity of the account—Did the researcher self consciously assess the likely impact of the methods used on the data obtained? Were sufficient data included in the reports of the study to provide sufficient evidence for readers to assess whether analytical criteria had been met?

Conclusion

Although the issue of quality in qualitative health and health services research has received considerable attention, a recent paper was able to argue, legitimately, that “quality in qualitative research is a mystery to many health services researchers.”
However, qualitative researchers can address the issue of quality in their research.
As in quantitative research, the basic strategy to ensure rigour, and thus quality, in qualitative research is systematic, self conscious research design, data collection, interpretation, and communication.
Qualitative research has much to offer. Its methods can, and do, enrich our knowledge of health and health care.
It is not, however, an easy option or the route to a quick answer.
As Dingwall et al conclude, “qualitative research requires real skill, a combination of thought and practice and not a little patience.”