Research Design

Science

a process of careful and systematic inquiry. The discovery of knowledge; what do we know and how do we know it

Research

a structured way of problem solving. A specific method used to discover that knowledge

• systematic and objective (based on sound evidence)

• interpretations of test scores are valid and reliable (consistent with what is being researched)

• dynamic and creative

• examines conditions and outcomes (sex, age, income)

• to improve methods of practice in an ethical way

Ways of knowing

• Intuition

• Authority

• Rationalism

• Empirism

• Scientific method

intuition

Relying on emotions, instincts or gut feeling of what feels right.

strength: quick

Drawback: often biased or wrong

Ex: I just feel like cold water reduces inflammation. so it must help revovery

Authority

accepting new ideas because an authority states they are true.

Strength: can come from someone of valuable knowledge

Drawback: can be an opinion or no evidence provided

Ex: my therepist says…

Rationalism

using logic and reasoning to acquire new knowledge based on premises

Ex: since inflammation causes soreness, less inflammation should be faster recovery

Empiricism

Acquiring knowledge through observation and experience

Ex: i tried an ice bath once and i felt less sore

Scientific method

a process of systematically collecting and evaluating evidence to test ideas and answer question. Ensures maximum objectivity and consistency

Strength: minimizes biases

Drawback: takes time and resources, generalizability

Ex: randomly assigning control groups and research groups, analyzing findings, replicating them and forming conclusions.

Variables of research

Quality (design)→ the extent to which a study’s design, implementations, and analysis minimizes (or considers) bias

Quantity→ Quantifiable measurement, sample size, as well as strength of findings from data analyses

Consistency→ the degree to which similar findings are reported from studies that have similar and different designs (replicability)

Research wheel

Theory.→ research questions hypotheses→ observations→empirical generalizations

Theories

A systematic set of interrelated statements intended to explain some aspect of social life (breakdown core concepts to help explain things). Explain recurring patterns, tested over and over, explain aggregates not individuals (large groups or broad experiences), state a probability, chance, or tendency, not an absolute casual relationship (not causal)

Hypotheses

→ a tentative statement about the relationship among variables

Qualitative vs Quantitative

Qualitative: research aims/questions Ex: to explore athletes’ perceptions of burnout

Quantitative: like a hypothesis. Ex: athletes who are anxious, will experience more burnout. Includes Independent and dependent variables:

• Independent: casual variable that produces the effect

• Dependent: the resulting effect/outcome variable (measurable)

Data collection

operationalization of concepts into variables. Requires identifying the appropriate indicators eg. how will u measure anxiety and burnout

Empirical Generalizations

→ Analysis of data leads to generalized statements about the findings

→ examine or replicate findings in a different context/population

→ link findings back to theory

deductive vs inductive approaching

Deductive: theory testing, take general info from a theory and use it to explain certain events (start with theory then test with data)

Inductive: Theory construction, collecting data and building a theory from that (start with data and build theory)

The cloud concept

Hitting a dead end or getting confusing results does not

mean you are bad at research. Entering "The Cloud" is a

normal structural phase of discovery, not a personal or

technical failure.

Bypassing our inner critic - When data deviates from the

plan, treat the anomaly as a creative lead rather than a

mistake to hide.

THE ‘CLOUD’ CONCEPT

A published paper vs reality

Acknowledging not just the results but the process that

brought you there.

Schinke et al main ideas

Authors argue that successful academic publishing is a skill that can be learned. Using these steps:

1. start with strong research question: original contribution to existing knowledge, practical

2. Choose the right journal: one that fits the research

3. Follow publication ethics

4. Make the manuscript tell a clear story

5. Be honest about limitations

6. Make practical implications specific: explain what should be done and how

7. Treat reviewer feedback as an opportunity

Grant and Booth main ideas

There is no single best type of review. The appropriate review depends on the purpose and research question. Researchers should also clearly explain how their review was conducted.

Review terminology is inconsistent so researchers should provide clear and transparent descriptions of their methods rather than relying on labels.

SALSA Framework (from grant and booth)

S- Search: does the review use one or multiple databases? Limited or comprehensive search?

A- Appraisal: does the researcher evaluate the quality of the studies found?

S- Synthesis: how are findings brought together?

A- Analysis: how are findings interpreted?

Ways to find research

Research library: search tool that helps you find relevant research

Research Database: stores research based on specific topics (ex: PubMed)

Research Journal: a publication that contains new research articles (Ex: Kinesiology Review)

Research Article: the actual manuscripts that researchers write to report findings

Types of research

1. Theoretical: also called conceptual, categorize and describe constructs, map relationships among them. Not collecting data

2. Empirical: qualitative, quantitative, mixed methods-collecting data (control groups, surveys, observations etc)

3. Reviews: are integrative, a summary of broad themes in literature, assessing quality of research and giving future research directions.

Hierarchy of evidence

1. Randomized controlled trials (RCTs): the gold standard. minimizes cofounded variables, unbiased, high quality, but can be unethical

2. Cohort studies: quality still hogh, group of individuals observed over time, multiple intervals, good for clinical application

3. Case control studies: often used for understanding characteristics of illness and health outcomes overtime

4. Cross-sectional studies or surveys: more common, only collect at one time point, time efficient but cannot determine cause and effect only correlation.

5. Case studies/reports: study specific individual or small population to understand them, risk of bias is higher

6. Mechanistic studies: chemical/biological based

7. Editorials/expert opinion: often invited to write these, personal biases common but still evidence based

Type depends on question being asked

Why write reviews?

1. Consolidate evidence: gathering findings to develop a stronger evidence base

2. Critical appraisals and resolving conflicts: evaluate quality of existing evidence, resolve competing schools of though or contradictory evidence

3. Identifying research gaps, developing new theories: map out what we know, the quality of research, and can move beyond summaries

4. Supporting decision-making: synthesizing literature can inform practice and policy decisions.

Review types

• Critical

• Literature

• Mapping

• Meta-analysis

• Rapid

• Scoping

• Systematic

• Umbrella

Critical review

aims to demonstrate that the writer has extensively researched and critically evaluated the quality the literature, typically resulting in a hypothesis or new model

Strength: evaluating value of body of work promote further testing

Limitation: not very structured, no formal assessment of quality (subjective)

Literature review

Provides an examination of recent or current literature, covering a wide range of topics at various levels of completeness

Strengths: easily implemented, summarizes nicely, avoids research duplications

Limitations: Least rigorous, no systematic approach to identifying literature, selection bias

Mapping review

Mapping out and categorizing existing literature to identify gaps in research to commission further reviews or primary research.

Strength: more structured, categorizes within a broader context to eliminate gaps and help refine

Limitations: time intensive, no quality appraisal

Meta-analysis

a technique that statistically combines multiple quantitative studies to provide a more precise estimate of a treatment effect

Strength: reduces bias, mathematical

Limitations: depends on quality of research

Rapid Review

what is already known about a policy or practice issues that uses systematic review methods but makes concessions to the depth of the process to meet shorter timeframes.

Strengths: timely and systematic

Limitations: overlooking articles creates bias

Scoping Review

preliminary assessment of the size and scope of available research literature, identifies nature and extent of evidence

Strengths: determines whether systematic review is needed

Limitations: cant be used to make final decisions abut policy

Systematic review

rigorous method that systematically searches for, appraises, and synthesizes research evidence, follows explicit guidelines to ensure transparency and replicability.

Strengths: very strict guidelines

Limitations: depends on previous research and lots of it, narrow in scope as it addresses very specific questions

Umbrella review

compiles evidence from multiple existing reviews into a single accessible document

Quality of work depends on quality of previous work

Reading research: parts of a research article

Abstract: key points, overview, can be structured or unstructured, usually 150-300 words

Introductions: what do we know about a topic, sets stage for reader, ends with purpose of study

Method: research design, who what where when and how, ethics, data collection, analysis approach

Results: key findings, quantitative (stats) and qualitative ( participant quotes)

Discussion: interprets findings, puts the findings into context with regard to what we already know, discuss strengths and limitations

References: giving credit, APA 7Th edition

Writing research

1. Research topic: what are u interested in studying, narrowly focused, be familiar with existing research on topic

2. Research problem: why is this study needed? describe context for the study and issues within the literature, take note of whats been done and what hasn’t to create your question. Should be challenging, important, and feasible

3. Literature search and review: what do we already know, a synopsis (what researchers know based on existing studies) and Key papers that justify your topic, narrate your problem, and identify contrasting perspectives

4. Study purpose, research questions and hypotheses: narrowing in on specifics

Types of research problems

• Descriptive: describing an event, phenomenon, condition, or circumstance

• Predictive: need to identify relationship among variables that may vary over time or across cases

• Explanation: make claims about cause and effect, or attempt to answer why events and behaviours happen

Key things to remember when doing a literature review

1. identify seminal (30 years or more ago) articles that are referenced in newer articles

2. Check number of times a paper has been cited

3. Assess quality of nwer articles

Assessing research quality

• how recent is the article? if more than 10 years old but not considered seminal, exclude it

• Where is it published? a reputable journal? Is it peer-reviewed?

• Consider author background and bias. Are the credentials and affiliations listed?

• Check references

Predatory journals

• Prioritize self interest, false or misleading info

• Poor quality, journals that accept articles for publication without performing quality checks

• Transparency, is the editor or board members verified? can you find contact info?

• Aggressive solicitation, often use repeated emails, soliciting research articles, and urgency in tone

Annotated bibliography

notes of the most important components from articles that will help you write a literature review. Includes citation, purpose, theory and methods, key findings, and how it links to your study

Alon main ideas

A good scientific problem should be feasible and interesting as well as being personally meaningful

Types of problems:

• Easy + low interest (low-hanging fruit)

• Difficult + low interest (hard and not valuable)

• Difficult + high interest (ground challenges)

• High feasibility + high interest (most desirable area)

Pareto Front: contains problems where no other problem is clearly better in both feasibility and interest. Depends on career stage

3 month rule: spend at least three months reading, discussing, and planning

“The cloud”: a period where things dont work out, assumptions break down, results dont make sense etc. can lead to a problem C that may be more interesting and feasible than original problem.

A→ detour→ the cloud→ C

FINER criteria for strong research questions

F- Feasible

I- Interesting

N- Novel

E- Ethical

R- Relevant

PICOT criteria for constructing research questions

P- Population/problem

I- Intervention/Indicator

C- Comparison

O- Outcome

T- Timeframe

The Purpose

also called the objectives/aims of your research. Should be clearly described and justified in the purpose statement. This is a minimum of one paragraph long. It summarizes the topic and goals of the specific research study.

Purpose statement should act as an umbrella statement. Begin broad introducing the topic, identify a need, make an argument why is it important, identify theories, and summarize the purpose at end.

Can be qualitative (intention to explore, discover, understand, describe) or Quantitative (intention to test, relate, compare, reduce) Can be both which can either be Concurrent (both going on at same time) or Sequential (one or other happens first, informing the next)

Unit of Analysis

the what and whom you want to study

could be individuals (roles, relationships, positions), Social groups (families, organizations, cities), or artifacts books, documents, buildings, social media)

If we dont do this, we commit two fallacies: Ecological Fallacy which is drawing conclusions about individuals based on observations of groups, and Individualistic Fallacy which is drawing conclusions about groups based on some individuals.

Research questions

Central questions that guide the study, typically 1-2 primary ones, open ended, do not specify expected answers

Use PIE: Populations (what people, objects, or events), Variable/Concept under Investigation(what influence on population), and Evaluation/Effect (what type of data will best provide a response)

Forming hypotheses

Two componets:

The NULL HYPOTHESIS states that there is no effect of treatment on the phenomenon under investigation.

The ALTERNATIVE HYPOTHESIS is supported if the nill hypothesis is false. This does not mean mean that it is ‘true’ but rather is supported for now and can undergo further testing.

These always oppose one another

Directionality

Directional hypotheses are used when researchers have evidence to support that there will be a positive or negative impact of the independent variable on the dependent (there is evidence in existing literature of a directional difference, want to make a causal inference, most used in clinical research)

Non-directional hypotheses state that there is a relationship between two variables but does not state what the relationship or difference may be (exploratory in nature, used when lack of evidence to support directional difference)

The P Value

The P-value is the probability of getting results at least as extreme as those observed, if the null hypothesis were true (no effect)

Small p-value (eg: .01)= luck would rarely do this so “just luck” isn’t a convincing explanation

Large p-value(eg: .40)= luck could easily do this, so we cant rule it out

If luck would produce the result less than 5% of the time, they call it statistically significant. Therefore, the p-value is not the chance that your hypothesis is true, how big or important the effect is, or proof, it simply is the probability of luck.

Errors

Type 1 Error or False Positive: when researchers make the decision that s manipulation has been successful when it isn;t

Type 2 Error or False Negative: when researchers make the decision that the manipulation failed when it actually worked

Statistical significance vs Practical significance

Practical significance is when the effect is big enough to make a real difference. It askes “does this matter?” Rather than “is it real?” like a P-Value

Cohen’s d:

• 0.2=small

• 0.5=medium

• 0.8=large

Despite a p-value not being significant, the results can still be practically signifiant

McHugh et al main ideas

study showed that for these urban Aboriginal youth, sport had multiple interconnected meanings:

1. Purpose: Understand what sport means to urban Aboriginal youth in Edmonton.

2. Participants: 15 Aboriginal youth, ages 12–15.

3. Method: Photovoice + talking circles.

4. Four themes:

   • Activities I've grown up playing

   • Having fun

   • Being with nature and others

   • Believing in yourself

5. Participants mainly described contemporary sports, not traditional Aboriginal activities.

6. Sport was viewed positively and holistically.

7. Sport was associated with physical, emotional, mental, social, and personal-development benefits.

8. The authors emphasize Aboriginal participation and control in research and programming.

9. Major limitations included limited participant involvement in analysis, one-time data collection, consent issues, and limited generalizability.

10. Photovoice was considered valuable because it gave youth an opportunity to actively share their own meanings of sport.

Tiller and Ekkekaksis main ideas

Main problem: Kinesiology has questionable research practices that can reduce the accuracy, reproducibility, and credibility of research.

Main cause: The academic system creates perverse incentives, especially:

• Publish-or-perish

• Funding competition

• Citation metrics

• Impact factors

• Career advancement

• Financial rewards

The authors argue that simply teaching researchers not to use QRPs is not enough. The academic incentive system itself needs to change so that researchers are rewarded for transparent, rigorous, high-quality science rather than simply producing more publications and citations.

Major Questionable research practices (QRP)

1. Publication bias

2. Misrepresenting exploratory research as confirmatory

3. HARKing/post hoc hypotheses

4. Excessive self-citation

5. Data fabrication/falsification

6. Overreliance on p-values

7. Multiple comparisons

8. Omitting effect sizes

9. Misreporting variance

10. p-hacking

11. Low statistical power

12. Cherry-picking outcomes

13. Ignoring placebo effects

Ethical Standards in research

Perhaps the most important responsibility of researchers

• Focused on respecting the rights of study participants and protecting them from

harm

• Research carries risks and benefits

• Ethics should not be viewed as a single event but an ongoing process

Ethics policy in Canada

Developed, in part, in response to historical practices of unethical treatment of people in research

• Internationally recognized examples are the ‘medical experiments’

⚬ Nazi researchers during WWII

⚬ Tuskegee experiment

⚬ Willowbrook experiment

• In Canada:

⚬ 1940-1950s Indigenous children were

denied healthcare when conducting

nutrition research (Mosby, 2013)

• Policies were developed to prevent future ethical violations and to ensure that all participants are respected and protected from harm in research

Tri-council policy statement: ethical conduct for researchers involving humans (TCPS 2)

Joint policy for federal funding research agencies

• Informed by three core principles:

Respect for Persons: Intrinsic value of human beings and the respect and consideration that they are due

Concern for Welfare: Quality of that person’s experience of life in all aspects

Justice: Obligation to treat people fairly and equitably

TCPS 2

By applying these core principles, researchers strive to achieve two important ethical standards:

Protection: Researchers provide the necessary protection of participants

Shared Benefits: Research needs to result in shared benefits - whereby research meets the needs of researchers and participants