Research Design

Science

a process of careful and systematic inquiry. The discovery of knowledge; what do we know and how do we know it

Research

a structured way of problem solving. A specific method used to discover that knowledge

• systematic and objective (based on sound evidence)

• interpretations of test scores are valid and reliable (consistent with what is being researched)

• dynamic and creative

• examines conditions and outcomes (sex, age, income)

• to improve methods of practice in an ethical way

Ways of knowing

• Intuition

• Authority

• Rationalism

• Empirism

• Scientific method

intuition

Relying on emotions, instincts or gut feeling of what feels right.

strength: quick

Drawback: often biased or wrong

Ex: I just feel like cold water reduces inflammation. so it must help revovery

Authority

accepting new ideas because an authority states they are true.

Strength: can come from someone of valuable knowledge

Drawback: can be an opinion or no evidence provided

Ex: my therepist says…

Rationalism

using logic and reasoning to acquire new knowledge based on premises

Ex: since inflammation causes soreness, less inflammation should be faster recovery

Empiricism

Acquiring knowledge through observation and experience

Ex: i tried an ice bath once and i felt less sore

Scientific method

a process of systematically collecting and evaluating evidence to test ideas and answer question. Ensures maximum objectivity and consistency

Strength: minimizes biases

Drawback: takes time and resources, generalizability

Ex: randomly assigning control groups and research groups, analyzing findings, replicating them and forming conclusions.

Variables of research

Quality (design)→ the extent to which a study’s design, implementations, and analysis minimizes (or considers) bias

Quantity→ Quantifiable measurement, sample size, as well as strength of findings from data analyses

Consistency→ the degree to which similar findings are reported from studies that have similar and different designs (replicability)

Research wheel

Theory.→ research questions hypotheses→ observations→empirical generalizations

Theories

A systematic set of interrelated statements intended to explain some aspect of social life (breakdown core concepts to help explain things). Explain recurring patterns, tested over and over, explain aggregates not individuals (large groups or broad experiences), state a probability, chance, or tendency, not an absolute casual relationship (not causal)

Hypotheses

→ a tentative statement about the relationship among variables

Qualitative vs Quantitative

Qualitative: research aims/questions Ex: to explore athletes’ perceptions of burnout

Quantitative: like a hypothesis. Ex: athletes who are anxious, will experience more burnout. Includes Independent and dependent variables:

• Independent: casual variable that produces the effect

• Dependent: the resulting effect/outcome variable (measurable)

Data collection

operationalization of concepts into variables. Requires identifying the appropriate indicators eg. how will u measure anxiety and burnout

Empirical Generalizations

→ Analysis of data leads to generalized statements about the findings

→ examine or replicate findings in a different context/population

→ link findings back to theory

deductive vs inductive approaching

Deductive: theory testing, take general info from a theory and use it to explain certain events (start with theory then test with data)

Inductive: Theory construction, collecting data and building a theory from that (start with data and build theory)

The cloud concept

Hitting a dead end or getting confusing results does not

mean you are bad at research. Entering "The Cloud" is a

normal structural phase of discovery, not a personal or

technical failure.

Bypassing our inner critic - When data deviates from the

plan, treat the anomaly as a creative lead rather than a

mistake to hide.

THE ‘CLOUD’ CONCEPT

A published paper vs reality

Acknowledging not just the results but the process that

brought you there.

Schinke et al main ideas

Authors argue that successful academic publishing is a skill that can be learned. Using these steps:

1. start with strong research question: original contribution to existing knowledge, practical

2. Choose the right journal: one that fits the research

3. Follow publication ethics

4. Make the manuscript tell a clear story

5. Be honest about limitations

6. Make practical implications specific: explain what should be done and how

7. Treat reviewer feedback as an opportunity

Grant and Booth main ideas

There is no single best type of review. The appropriate review depends on the purpose and research question. Researchers should also clearly explain how their review was conducted.

Review terminology is inconsistent so researchers should provide clear and transparent descriptions of their methods rather than relying on labels.

SALSA Framework (from grant and booth)

S- Search: does the review use one or multiple databases? Limited or comprehensive search?

A- Appraisal: does the researcher evaluate the quality of the studies found?

S- Synthesis: how are findings brought together?

A- Analysis: how are findings interpreted?

Ways to find research

Research library: search tool that helps you find relevant research

Research Database: stores research based on specific topics (ex: PubMed)

Research Journal: a publication that contains new research articles (Ex: Kinesiology Review)

Research Article: the actual manuscripts that researchers write to report findings

Types of research

1. Theoretical: also called conceptual, categorize and describe constructs, map relationships among them. Not collecting data

2. Empirical: qualitative, quantitative, mixed methods-collecting data (control groups, surveys, observations etc)

3. Reviews: are integrative, a summary of broad themes in literature, assessing quality of research and giving future research directions.

Hierarchy of evidence

1. Randomized controlled trials (RCTs): the gold standard. minimizes cofounded variables, unbiased, high quality, but can be unethical

2. Cohort studies: quality still hogh, group of individuals observed over time, multiple intervals, good for clinical application

3. Case control studies: often used for understanding characteristics of illness and health outcomes overtime

4. Cross-sectional studies or surveys: more common, only collect at one time point, time efficient but cannot determine cause and effect only correlation.

5. Case studies/reports: study specific individual or small population to understand them, risk of bias is higher

6. Mechanistic studies: chemical/biological based

7. Editorials/expert opinion: often invited to write these, personal biases common but still evidence based

Type depends on question being asked

Why write reviews?

1. Consolidate evidence: gathering findings to develop a stronger evidence base

2. Critical appraisals and resolving conflicts: evaluate quality of existing evidence, resolve competing schools of though or contradictory evidence

3. Identifying research gaps, developing new theories: map out what we know, the quality of research, and can move beyond summaries

4. Supporting decision-making: synthesizing literature can inform practice and policy decisions.

Review types

• Critical

• Literature

• Mapping

• Meta-analysis

• Rapid

• Scoping

• Systematic

• Umbrella

Critical review

aims to demonstrate that the writer has extensively researched and critically evaluated the quality the literature, typically resulting in a hypothesis or new model

Strength: evaluating value of body of work promote further testing

Limitation: not very structured, no formal assessment of quality (subjective)

Literature review

Provides an examination of recent or current literature, covering a wide range of topics at various levels of completeness

Strengths: easily implemented, summarizes nicely, avoids research duplications

Limitations: Least rigorous, no systematic approach to identifying literature, selection bias

Mapping review

Mapping out and categorizing existing literature to identify gaps in research to commission further reviews or primary research.

Strength: more structured, categorizes within a broader context to eliminate gaps and help refine

Limitations: time intensive, no quality appraisal

Meta-analysis

a technique that statistically combines multiple quantitative studies to provide a more precise estimate of a treatment effect

Strength: reduces bias, mathematical

Limitations: depends on quality of research

Rapid Review

what is already known about a policy or practice issues that uses systematic review methods but makes concessions to the depth of the process to meet shorter timeframes.

Strengths: timely and systematic

Limitations: overlooking articles creates bias

Scoping Review

preliminary assessment of the size and scope of available research literature, identifies nature and extent of evidence

Strengths: determines whether systematic review is needed

Limitations: cant be used to make final decisions abut policy

Systematic review

rigorous method that systematically searches for, appraises, and synthesizes research evidence, follows explicit guidelines to ensure transparency and replicability.

Strengths: very strict guidelines

Limitations: depends on previous research and lots of it, narrow in scope as it addresses very specific questions

Umbrella review

compiles evidence from multiple existing reviews into a single accessible document

Quality of work depends on quality of previous work

Reading research: parts of a research article

Abstract: key points, overview, can be structured or unstructured, usually 150-300 words

Introductions: what do we know about a topic, sets stage for reader, ends with purpose of study

Method: research design, who what where when and how, ethics, data collection, analysis approach

Results: key findings, quantitative (stats) and qualitative ( participant quotes)

Discussion: interprets findings, puts the findings into context with regard to what we already know, discuss strengths and limitations

References: giving credit, APA 7Th edition

Writing research

1. Research topic: what are u interested in studying, narrowly focused, be familiar with existing research on topic

2. Research problem: why is this study needed? describe context for the study and issues within the literature, take note of whats been done and what hasn’t to create your question. Should be challenging, important, and feasible

3. Literature search and review: what do we already know, a synopsis (what researchers know based on existing studies) and Key papers that justify your topic, narrate your problem, and identify contrasting perspectives

4. Study purpose, research questions and hypotheses: narrowing in on specifics

Types of research problems

• Descriptive: describing an event, phenomenon, condition, or circumstance

• Predictive: need to identify relationship among variables that may vary over time or across cases

• Explanation: make claims about cause and effect, or attempt to answer why events and behaviours happen

Key things to remember when doing a literature review

1. identify seminal (30 years or more ago) articles that are referenced in newer articles

2. Check number of times a paper has been cited

3. Assess quality of nwer articles

Assessing research quality

• how recent is the article? if more than 10 years old but not considered seminal, exclude it

• Where is it published? a reputable journal? Is it peer-reviewed?

• Consider author background and bias. Are the credentials and affiliations listed?

• Check references

Predatory journals

• Prioritize self interest, false or misleading info

• Poor quality, journals that accept articles for publication without performing quality checks

• Transparency, is the editor or board members verified? can you find contact info?

• Aggressive solicitation, often use repeated emails, soliciting research articles, and urgency in tone

Annotated bibliography

notes of the most important components from articles that will help you write a literature review. Includes citation, purpose, theory and methods, key findings, and how it links to your study

Alon main ideas

A good scientific problem should be feasible and interesting as well as being personally meaningful

Types of problems:

• Easy + low interest (low-hanging fruit)

• Difficult + low interest (hard and not valuable)

• Difficult + high interest (ground challenges)

• High feasibility + high interest (most desirable area)

Pareto Front: contains problems where no other problem is clearly better in both feasibility and interest. Depends on career stage

3 month rule: spend at least three months reading, discussing, and planning

“The cloud”: a period where things dont work out, assumptions break down, results dont make sense etc. can lead to a problem C that may be more interesting and feasible than original problem.

A→ detour→ the cloud→ C

FINER criteria for strong research questions

F- Feasible

I- Interesting

N- Novel

E- Ethical

R- Relevant

PICOT criteria for constructing research questions

P- Population/problem

I- Intervention/Indicator

C- Comparison

O- Outcome

T- Timeframe

The Purpose

also called the objectives/aims of your research. Should be clearly described and justified in the purpose statement. This is a minimum of one paragraph long. It summarizes the topic and goals of the specific research study.

Purpose statement should act as an umbrella statement. Begin broad introducing the topic, identify a need, make an argument why is it important, identify theories, and summarize the purpose at end.

Can be qualitative (intention to explore, discover, understand, describe) or Quantitative (intention to test, relate, compare, reduce) Can be both which can either be Concurrent (both going on at same time) or Sequential (one or other happens first, informing the next)

Unit of Analysis

the what and whom you want to study

could be individuals (roles, relationships, positions), Social groups (families, organizations, cities), or artifacts books, documents, buildings, social media)

If we dont do this, we commit two fallacies: Ecological Fallacy which is drawing conclusions about individuals based on observations of groups, and Individualistic Fallacy which is drawing conclusions about groups based on some individuals.

Research questions

Central questions that guide the study, typically 1-2 primary ones, open ended, do not specify expected answers

Use PIE: Populations (what people, objects, or events), Variable/Concept under Investigation(what influence on population), and Evaluation/Effect (what type of data will best provide a response)

Forming hypotheses

Two componets:

The NULL HYPOTHESIS states that there is no effect of treatment on the phenomenon under investigation.

The ALTERNATIVE HYPOTHESIS is supported if the nill hypothesis is false. This does not mean mean that it is ‘true’ but rather is supported for now and can undergo further testing.

These always oppose one another

Directionality

Directional hypotheses are used when researchers have evidence to support that there will be a positive or negative impact of the independent variable on the dependent (there is evidence in existing literature of a directional difference, want to make a causal inference, most used in clinical research)

Non-directional hypotheses state that there is a relationship between two variables but does not state what the relationship or difference may be (exploratory in nature, used when lack of evidence to support directional difference)

The P Value

The P-value is the probability of getting results at least as extreme as those observed, if the null hypothesis were true (no effect)

Small p-value (eg: .01)= luck would rarely do this so “just luck” isn’t a convincing explanation

Large p-value(eg: .40)= luck could easily do this, so we cant rule it out

If luck would produce the result less than 5% of the time, they call it statistically significant. Therefore, the p-value is not the chance that your hypothesis is true, how big or important the effect is, or proof, it simply is the probability of luck.

Errors

Type 1 Error or False Positive: when researchers make the decision that s manipulation has been successful when it isn;t

Type 2 Error or False Negative: when researchers make the decision that the manipulation failed when it actually worked

Statistical significance vs Practical significance

Practical significance is when the effect is big enough to make a real difference. It askes “does this matter?” Rather than “is it real?” like a P-Value

Cohen’s d:

• 0.2=small

• 0.5=medium

• 0.8=large

Despite a p-value not being significant, the results can still be practically signifiant

McHugh et al main ideas

study showed that for these urban Aboriginal youth, sport had multiple interconnected meanings:

1. Purpose: Understand what sport means to urban Aboriginal youth in Edmonton.

2. Participants: 15 Aboriginal youth, ages 12–15.

3. Method: Photovoice + talking circles.

4. Four themes:

   • Activities I've grown up playing

   • Having fun

   • Being with nature and others

   • Believing in yourself

5. Participants mainly described contemporary sports, not traditional Aboriginal activities.

6. Sport was viewed positively and holistically.

7. Sport was associated with physical, emotional, mental, social, and personal-development benefits.

8. The authors emphasize Aboriginal participation and control in research and programming.

9. Major limitations included limited participant involvement in analysis, one-time data collection, consent issues, and limited generalizability.

10. Photovoice was considered valuable because it gave youth an opportunity to actively share their own meanings of sport.

Tiller and Ekkekaksis main ideas

Main problem: Kinesiology has questionable research practices that can reduce the accuracy, reproducibility, and credibility of research.

Main cause: The academic system creates perverse incentives, especially:

• Publish-or-perish

• Funding competition

• Citation metrics

• Impact factors

• Career advancement

• Financial rewards

The authors argue that simply teaching researchers not to use QRPs is not enough. The academic incentive system itself needs to change so that researchers are rewarded for transparent, rigorous, high-quality science rather than simply producing more publications and citations.

Major Questionable research practices (QRP)

1. Publication bias

2. Misrepresenting exploratory research as confirmatory

3. HARKing/post hoc hypotheses

4. Excessive self-citation

5. Data fabrication/falsification

6. Overreliance on p-values

7. Multiple comparisons

8. Omitting effect sizes

9. Misreporting variance

10. p-hacking

11. Low statistical power

12. Cherry-picking outcomes

13. Ignoring placebo effects

Ethical Standards in research

Perhaps the most important responsibility of researchers

• Focused on respecting the rights of study participants and protecting them from

harm

• Research carries risks and benefits

• Ethics should not be viewed as a single event but an ongoing process

Ethics policy in Canada

Developed, in part, in response to historical practices of unethical treatment of people in research

• Internationally recognized examples are the ‘medical experiments’

⚬ Nazi researchers during WWII

⚬ Tuskegee experiment

⚬ Willowbrook experiment

• In Canada:

⚬ 1940-1950s Indigenous children were

denied healthcare when conducting

nutrition research (Mosby, 2013)

• Policies were developed to prevent future ethical violations and to ensure that all participants are respected and protected from harm in research

Tri-council policy statement: ethical conduct for researchers involving humans (TCPS 2)

Joint policy for federal funding research agencies

• Informed by three core principles:

Respect for Persons: Intrinsic value of human beings and the respect and consideration that they are due

Concern for Welfare: Quality of that person’s experience of life in all aspects

Justice: Obligation to treat people fairly and equitably

TCPS 2

By applying these core principles, researchers strive to achieve two important ethical standards:

Protection: Researchers provide the necessary protection of participants

Shared Benefits: Research needs to result in shared benefits - whereby research meets the needs of researchers and participants

Research at universities

• Research Ethics Boards: Provide guidance and approval for research

• Human Research Ethics Policies: Anyone conducting research must adhere

• Biohazard approvals: biological or medical research also requires biohazard approval

Key guidelines of uni research

1. Conflicts of Interest

2. Process of consent

3. Privacy and confidentiality

Conflicts of interest

Study question and Purpose: when a study puts researchers in a conflict with their research, personal, or institutional interests (duties).

Economic Investments: is the researcher a shareholder (a part of the money sitch) in the organization (uni) that wants to fund the study?

Interpersonal Relationships: Is the researcher related to any of the participants or end-users involved with the data collection process?

Resolusion?

• Must disclose real, potential, or perceived conflicts of interest to REB and they will determine next steps

Process of consent

1. must ensure participants’ free, informed, and ongoing consent throughout the entire research process.

2. Consent shall be voluntary

3. Consent can be withdrawn at any time

4. If participants withdraw consent, they can also request to withdraw their data

5. Incentives impact voluntariness

• important to also keep in mind: Participants who lackj capacity to consent (children, disabilities) and Partial disclosure or deception (demonstrating minimal risk, it is necessary, and requires debriefing)

Privacy and confidentiality

A persons privacy refers to the right to be free from intrusion by others. Must be protected across all research phases.

Confidentiality is the researchers obligation to safeguard entrusted information

There are various ways to safeguard: participant codes, pseudonyms, locked locations, and password protected files

Research with Indigenous peoples

Root of colonialist practices: research has resulted in exploitation and distrust

Upholding Justice: considering the power imbalances between Indigenous Peoples and researchers

Community Engagement: creating research with end-users rather than “for” or “on” them

Customs of community: researchers responsibility to become informed about and adhere to codes of practice

OCAP Principles: ownership, control, access, and possession

Principles in Action

An exploration of meanings of sport to urban Aboriginal Youth

• Purpose: to explore the meanings of sport to urban indigenous youth

• Collaborative Process: the research team engaged Indigenous community partners throughout with a focus on building genuine relationships

• Community Customs: Community consultations were held via sharing circles. Those who facilitated the sharing circles were offered Tobacco to acknowledge and support cultural traditions as well as feasts of meat, cheese, fruit, and veggies offered to all participants.

Animal research Ethics

Animal Care Ethics Committees must obtain appropriate approval

3 R Tenet:

• Replace: avoid or replace the use of animals wherever possible

• Reduce: employ strategies that will result in fewer animals being used and which are consistent with experimental design

• Refine: modify experimental procedures to minimize pain and stress

It is the responsibility of the researcher to ensure that animals are only used when best efforts to identify an alternative have been explored

Ethical Decision-Making

• Professionals need ethical decision-making processes to help minimize damage and maximize benefit to the community, profession, and a professional’s career

• An Ethical Dilemma is when there is no clearly correct ethical choice. Its a situation that requires action, but the professional is required to two or more mutually exclusive actions (if they do one thing, cant do other)

Ethical Decision-Making process for professionals

1. Develop ethical sensitivity (choices we make affect others)

2. Consult code of ethics' (is there a conflict of interest)

3. Search ethics literature (useful when a profession’s code of conduct does not adequately address specific elements of situation requiring ethical decision making)

4. Develop self-awareness (reflect on the bias you bring to a situation that could impact ur decision making)

5. Apply ethics principles (VAJ Not Butt Fuck)

6. Develop alternative course for action (detail probable costs and benefits of each possible action)

7. Consult with colleagues (gain diff perspectives, objective lens, maintain privacy)

8. Take action (implement chosen plan, inform those affected and assume responsibility, document decision)

9. Reflect upon results of actions

Ethics Principles

VAJ Not Butt Fuck

V- Veracity (truthfulness)

A- Autonomy (freedom to make one’s own choice and take actions based on one’s own personal values

J- Justice (fair, equitable, and appropriate treatment of others)

N- Nonmaleficence (Obligation to not inflict harm upon others intentionally)

B- Beneficence (obligation to act for the benefit of others)

F- Fidelity (fulfilling one’s responsibilities of trust)

The ostrich effect

• A cognitive bias wherein people tend to “bury their head in the sand” and avoid or ignore potentially negative but useful information, just to avoid psychological discomfort.

• Kin and related fields often exhibit a reluctance to acknowledge widespread methodological shortcomings despite high-profile cases of misconduct

• Misconduct vs Questionable Research Practices (QRPs)

  • Misconduct involves blatant fraud like data falsification

  • QRPs are more subtle, deep-rooted practices introduced deliberately or inadvertently into a study leading to non-replicable results and conclusions

These practices increase frequency of false positives, diminish scientific rigor and erode public trust

Common QRPs

Publication bias: researchers are less likely to submit negative or neutral results

HARKing: “Hypothesizing After the Results are Known”. When a hypothesis is generated retroactively after data analysis but is presented as if it were designed prior

Statistical Issues: over-reliance on p values, low statistical power, omitting effect sizes

Drivers of Misconduct

Publish or Perish Culture: academic system prioritizes quantitative metrics over quality, incentivizing researchers to produce high-volume, low-quality output to “survive in academia”

Financial and corporate influence: industry-funded research may be more likely to report favourable outcomes for sponsored products, such as companies downplaying the role of diet in weight management.

Predatory Journals

Lack of Robust Peer Review

• academic journals that do not employ high-quality peer review processes

• faster publication rates and low publication fees

Supporting Publish or Perish

• fueled by extreme publication pressures

• appearing at rates that now exceed reputable journals

Publication Inflation

• authors trying to inflate their number of publications

• taking a calculated risk

Low Impact

• 60% of articles published in predatory journals recieve no citations in their first 5 year

Recommendations for Reform

Pre-registration→ registering hypotheses and methods before data collection to prevent “cherry-picking” outcomes

Robust Reporting→ mandating the reporting of effect sizes, confidence intervals, and power calculations

Ethics Education→ strengthening research ethics and statistical training in higher education

Nixon 2019 main idea

Nixon argues that health inequities cannot be eliminated by focusing only on disadvantaged populations. Meaningful change requires recognizing privilege, understanding how systems of inequality operate, and working collectively to dismantle those systems.

Messick 1995 main ideas

explains the importance of validity when evaluating educational and psychological assessments, especially performance assessments.

Messick concludes that the six aspects of construct validity must be considered together when evaluating assessments.

Validity is a unified concept that requires multiple forms of evidence to support the interpretation and use of scores.

Diaz et al main ideas

The researchers wanted to investigate whether short movement breaks could improve psychosocial outcomes in adults who spend extended periods sitting.

Study design: The Body Electric Study was a large-scale, pragmatic, non-randomised intervention.

Self-reported surveys, daily diaries, and momentary assessments were used to collect information.

4. Main Findings

Meredith et al main ideas

How sport and exercise psychology (SEP) research has measured behaviour and used different research methods over time.

• Researchers used a systematic coding process to classify the measures and research methods used in each study. Looked at 4 different journals

• Different research methods provide different types of information.

  • Quantitative methods provide numerical data and allow researchers to identify patterns.

  • Qualitative methods provide more detailed information about personal experiences and individual differences.

  • Mixed methods combine both approaches to provide a more complete understanding.

  The authors support using a variety of methods depending on the research question.