Research Methods Lectures
Research Methods
Why Science?
We have biases and errors
Science makes it as objective as possible (can never be fully objective)
Empiricism
We as humans have a lot of biases and errors. In fact, when we think we are trusting our intuition, oftentimes we can be quite wrong. In Fact, there’s something called the overconfidence phenomenon, where the more confident you are, actually probably the more wrong you are. We can’t really trust our intuition always.
Some examples include smart mathematicians from the late 1800s saying things like flying machines are impossible. They were super confident in saying that, but they were wrong.
Correlations between things that don’t actually exist. For example, more crime during a full moon, or every time you get your car washed, it rains. There is no data to support these claims. We make correlations most of the time based on what we want to believe.
The whole purpose of science is to hopefully be as objective as possible. Biology and chemistry aren’t purely objective either. It’s run by human beings. There are always ways of putting in human bias.
The whole point of science is to remove as many biases as possible and to be as objective as possible.
Our task is to look through studies that have been done and science to make sure that we’re following this idea of being objective. It takes critical analysis on our part. We can’t just say that a study was run, so therefore it’s completely true. We’re looking for a collection of things that tell us a study is true.
We’re focusing on empiricism, which is basically using a systematic approach to finding knowledge, so learning based on observation. This is not just something someone feels but something we can witness and see.
Examining it, replicating it, and analyzing it in a very thorough, systematic way.
The Scientific Method
The whole idea is that you’re not just going willy-nilly to see what happens or making all kinds of observations without some focus. The point of the scientific method is to have a focused question, called a hypothesis, and then test it.
Hopefully, what happens is you start with some sort of theory that comes from some observations. That theory helps guide what your predictions will be and what it will mean if your predictions are supported or not supported.
For example, I could go out there and observe that men are physically stronger than women on average. Where do you go from there? You need a theory.
One’s theory might be that testosterone creates more muscle fibers, and therefore men who have higher levels of testosterone are more likely to have higher levels of muscle tissue.
Now that I have that theory, I have some specifics I can go and test. I can double-check to see if, in fact, men have more muscle tissue than women and if they have more testosterone than women do.
If I find that the muscle tissue is different, that supports my theory. If I find that muscle tissue is no different between males and females, well then time to get a new theory or to modify that one quite a bit.
The theory helps guide how I tested, and I’m actually going to go in and physically measure their muscle tissue. My next step might be to test testosterone levels. And I can do an experiment, administer testosterone to women, and see if it changes their muscle tissue. Spoiler alert: it does.
We know that testosterone has something to do with it. My theory might be that it’s training that women get less exposure to things that are muscle-building.
If that’s my hypothesis, I’ll do an experiment where I make women exercise harder to see what happens.
The point of the theory is to make very specific and testable hypotheses and then help us interpret those.
I come up with a theory; now I make observations to create a hypothesis. I may run an experiment, which then will create more observations, hypotheses, etc. This will have a lot to say about my theory; my theory may need to be modified, or maybe it supports my theory. Or maybe it says, man, this theory is so far off we have to get rid of it altogether.
Not just a random idea, they’re backed up by all the tests that we’ve done. This isn’t any random person’s idea. A theory has a lot of stuff underneath it and a lot of support for it.
If I do enough observations, hypotheses, and experiments, then at that point I keep finding the same thing over and over and over again, we might call it a law.
For example, the law of gravity.
Theory is guided by very specific hypotheses, making observations and constantly revising the process. This is what makes science science.
Types of Research
Types of Research
Quantitative vs Qualitative
Quantitative
Method favored by psychologists. This is the fundamental cornerstone of psychology research
This is a little more easily interpreted. And allows us to make more inferences.
This is usually the favored method.
It’s going to have numbers associated with it. We’ve aggregated the data down to numerical information. We can run correlations; we can run ANOVAs; we can look at mean differences, which allows us to make mathematical inferences.
Can’t overstate what they’ve found. They can’t talk to the nuances behind it. They’re limited to the data.
Qualitative
In counseling, for example, most of the research tends to be more qualitative.
This information tends to be more detailed and rich.
This is going to include a lot of things like case studies and quotes from people, and devising themes that you might see.
Can’t make inferences and guess about the data. Can’t make statements about the general population.
These feed each other. A really good qualitative study is going to be deeply rich, and it’s going to help me formulate questions I can ask in a more numerical way. Good quantitative research really can benefit from some qualitative research that fleshes out the gaps.
Types of Research
Quantitative vs Qualitative
Descriptive correlational vs Experimental / Quasi
Descriptive: describing what you see. Asking a big group of people about things such as age or race. I can describe that 50% of the group is this, while 60% of the group is that. This doesn’t tell us much or help us predict.
Correlational study: which allows me to take two of those variables that are descriptive, and I just check to see if they hang together. If one rises, does the other move as well? Does it rise up, or does it go down? If they’re moving in the same direction, that gives you a sense that they go together.
When people’s stress levels rise, so does their depression level. They seem to be moving together. If you have less stress, you have less depression. That tells me that they are related in some way.
Correlation does not equal causation.
Just because two things seem to be related, I don’t know what caused what. I can’t say that stress causes depression. Nor can it be said that depression causes stress. Something else entirely could be causing both stress and depression.
Quasi-experimental study: The idea behind this type of study is that there are two or more groups that are already pre-separated.
For example, taking a bunch of cigarette smokers and non-cigarette smokers and looking at their long-term health over time. This is a great design, built to follow both groups for 30 years. Is there a difference between them?
The reason this isn’t the gold standard is that people put themselves in those groups. It’s possible the smoking is causing the lung damage over time, or the event that caused a person to smoke could be leading to the lung damage over time.
A true experiment would be: I take a whole bunch of people who don’t smoke. I divide them in half: half of you smoke three packs a day, half of you don’t. Let’s see what happens to your lungs. That would be the maximum amount of control you can have. A lot of ethical issues with that.
We have to rely on quasi-experiments for things like smoking or gender studies. This can be very useful information and more convincing than correlational. Even though the people sorted themselves into groups, I can show that lung damage doesn’t cause smoking. The smoking came first, so you can eliminate one possibility, but it still doesn’t prove that something else entirely is leading them to smoke and causing lung damage.
Experimental: is our absolute gold standard.
I put you into the groups. I divide you into groups. Each group is treated a little differently. And then we check to see what happens over time.
With stress and depression, we’ve noticed a correlation. Ethically, we can’t stress people out too badly, but we do have methods of stress that we can use.
Doing math problems quickly or putting your hand in freezing ice-cold water.
We can check to see if that has an impact on your mood.
The key is that I have two groups that are randomly assigned to two different conditions. In theory, both of those groups look kind of the same because it was all randomized. Qualities like smartness and motivation, hopefully, are divided equally between the groups.
Seeing if anything is different. If the stressed-out group suddenly has a negative mood and my non-stressed-out group is fine, that’s pretty convincing.
The experiment is gold. The two major problems. One: expensive; it’s really expensive to run an experiment. More importantly, it’s unethical sometimes. We can’t do an experiment for everything.
Smoking causes cancer, though. Cigarette companies love to tell you that it isn’t proven that cigarettes cause lung cancer. Technically, they’re right because we can’t do experiments.
Some many correlational and quasi-experimental studies and animal studies. It’s not a question that cigarettes cause lung damage.
Types of Research
Quantitative vs Qualitative
Descriptive vs Correlational
Three basic outcomes with a correlational analysis.
I could find out that they are positively correlated. Both go up together.
Height and weight. As your height goes up, so does your weight.
I could have a finding that is negatively correlated. As one goes up, the other goes down.
Sleep and tiredness. The more sleep a person gets, the less tired they will be. Productivity and Netflix hours viewed. Getting less down when binging a new Netflix show. As Netflix watching goes up, productivity goes down.
There is the possibility of no correlation at all.
There is no correlation between shoe size and hours of sleep.
Types of Research
Quantitative vs Qualitative
Descriptive vs Correlational vs Experimental/ Quasi
With a correlation, I don’t know what caused what.
It could be that X causes Y. Y could be causing X. Z could be causing both X and Y.
Experiments: If I’m going to run an experiment, then first I need to put people in two groups. The first thing I have to ask myself is: Did I randomly assign those groups?
If the groups are random, then it’s a true experiment or a randomized experiment.
If the groups aren’t random, then I ask: Is there a control group or multiple measures? If yes, quasi-experiment. If no, then there is no experiment. This would be a description.
Types of Research
Quantitative vs Qualitative
Descriptive vs Correlational vs Experimental/ Quasi
Longitudinal vs cross-sectional
These are usually more correlational studies, or quasi-experiments can be done this way too.
A cross-sectional study says: I do everything at once. I might have three different groups that I’m looking at, and I look to see if they are different in some way. Looking at seniors vs. juniors vs. sophomores in high school. Maybe I follow them for a week or two and I take their data down and I’m looking to see do seniors look different than juniors look different than sophomores?
We call this a cross-sectional study becayse we’re taking a snapshot in time but it doesn’t tell us much about these groups over time.
Maybe I find out that the seniors are the most depressed, juniors are a little less depressed and sophomores are veen less depressed. I might be tempted to come up with a conclusion that it just really sucks to be a senior. But I’m using this out of the context of is going on in their life at the time.
If this is 2020, then the seniors in that group missed out on prom and graduation. Does it have anything to do with being a senior? No, it’s in that context.
Longitudinal Studies: We follow groups over time.
If our theory is that the farther up in high school you go, the more depressed you become, then I want to start you as a freshman, measure you every year to see if you are getting more depressed the closer you get to senior year. And I’m going to do that with multiple cohorts. So, every year I’m going to take a new group of freshmen, and I’m going to follow them for several years to see what happens. This is going to give me a much better sense of my theory in that way.
How we collect data
Observations
I’m going to watch you and write things down. I could find a correlation this way. It’s not an experiment; it’s probably going to be descriptive, at best, correlational.
Case Studies
Genie, who was a little girl. She was neglected pretty severely in her home. She was forced to live in a basement and did not have any interaction with humans for the most part. They really checked in on her language development. There was a really thorough case study. A lot of different psychologists took numerous observations and notes in a very detailed way.
Because this is about one person, this is very rich. Case studies can be about one or maybe a couple of people. They are very detailed and very rich. But case studies aren’t necessarily applicable. We don’t know if we can take what we learned from Genie and apply it to anyone else. But it is rich, and it gives us a lot of ideas to go on and ways to make theories and hypotheses.
One of the hypotheses from this is that there is a very sensitive period of time where you learn language. And if you miss that boat, you miss the opportunity to really develop language. This is still being debated quite a bit.
Naturalistic or Laboratory
More short-term and can do lots of observations of different people with teens or even kids. You might go to schools and observe and take notes about people’s behavior.
Naturalistic means I do it in the setting where they are typically comfortable. Home, school, recess, things like that.
Laboratory means I bring you into a setting that I’ve created to observe you.
There are some advantages to both of these.
In a laboratory study, I’m controlling everything. If I really want to know how teens react to feedback on an essay. I want to see if giving them bad feedback affects them. Does it affect their self-esteem?
I can bring them into this laboratory study, and nothing else is going on. I’m only presenting them with this essay data. Then I can see how they react to that.
In the real world, they may receive bad feedback from a teacher. But there are other factors like friends and note passing. It’s hard to narrow it down to what I actually want to pay attention to.
If I want to see if kids are aggressive and I go to school, there are provocations happening. There’s back history we don’t know about. There’s the temperature of that day might be really hot, so everybody’s extra aggressive. There are all these extra things I can’t control in the real world.
The laboratory is a lovely controlled setting. The problem is you really know you’re being observed in a lab. This is unlike real life. So there are a couple of big problems.
One is called the Hawthorne Effect. You’d change when you’re being watched. This is even true of roaches, ants, and atoms, which behave differently when being observed.
Also, we don’t know how it will look in the real world. They may behave this way in a lab. But is that typical of their home life?
Naturalistic studies help with the Hawthorne effect, but it also gives us more realistic or natural results, which are going to be harder to interpret. And the Hawthorne effect doesn’t completely go away.
Going to an elementary school class, walking into a room, and being the most interesting thing in the room. You can’t use hidden cameras because those are unethical.
We won’t throw away all research because of this effect, and we have a lot of data that says if you just stick with it, people get used to the camera, or you and they forget it’s there, so you just have to give it some time.
Surveys
Another is surveys, we can ask you questions about yourself. This is basically self-reporting. This is the cheapest and easiest thing you can do. The problem is it’s really coming from your self-report, which includes biases, misunderstandings, and could be outright lies. This method may not be super accurate.
For example, if you give a survey about racism. People will report that they’re not super racist. But if you follow them around in real life, you would see a lot of instances of racist behavior.
Our surveys do not necessarily reflect reality. It’s an important tool; I wouldn’t throw it away, but you want to take things with a grain of salt.
For example, when they do studies on sexuality and estimate how many people are in the different categories. Take it with a grain of salt; the numbers of alternative sexualities are probably higher than you think because people don’t want to admit it to themselves. They may not be lying; they may just be in denial.
Archival/ Content Analysis
Data or Narratives
You basically find data that already exists, or you’ll find other people’s records of things.
For example, do hurricanes lead to divorce? Is there a correlation there? You might go to different cities where there was a hurricane and look at divorce rates pre- and post natural diaster. You’re not collecting any data; you’re finding what’s already there.
Big studies that are done with multi-million-dollar grants will post all their data on the internet so everyone can use that data and check for correlations between variables.
The problem is you are capitalizing on what’s already there, and they might not have asked the questions you wanted. This can be limiting.
Assessments/ Tests/ Biological Info
Assessments and Tests: This could use something like the MMPI-2 (a huge personality and psychopathology questionnaire. It has 500 questions and has been validated) and be more confident in the results.
Seeing if a person has bipolar disorder by giving them a measure of bipolar symptomatology and then using that in a study.
Biological: I test things like heart rate or blood pressure. Testing stress levels and seeing if children have more stress than adults. Do they have more adrenaline in their system?
Looking for actual concrete biological information can be great too.
Reliability/ Validaity and Other Considerations
Reliability and Validity
Anytime we use a test or an assessment of some sort, we need to verify that it’s working the way we think it should be working.
We want it to be reliable, which means that the results are consistent. If I have an intelligence test and every time I give it to you have a different level of intelligence, then it’s not a great measure. It needs to be consistent unless you have changed. But for most people, it should be consistent.
The reason is that if you had a huge change, it could indicate that something is wrong, such as having a stroke.
Validity means it’s measuring what it’s supposed to measure; it’s accurate.
A great example of invalidity: when we used to measure depression, we also used questions like I feel jittery, I feel upset frequently, “ and “ my heart races. Those questions don’t measure depression because those aren’t depression symptoms; they’re anxiety symptoms.
When we split these questions into depression and anxiety, the tests became more accurate.
We want to be accurate and consistent.
I can be consistent and wrong. It’s possibly to be reliably wrong, but it’s not possible to be accurate and inconsistent. If you’re accurate and inconsistent, then you’re only sometimes right.
Ex: seeing a clock and it reads 2 o’clock, and it is 2 o’clock, but every other time you check the clock it still says 2 o’clock; this isn’t an accurate clock.
I can be reliable but not valid. But I can not be valid without being reliable.
Considerations
Random Assignment
We want an experiment to have random assignments.
Ex: New job search seminar, where I help people get jobs. I go to the local unemployment office and post all these posters that say come to my seminar on Saturday at 7 am. I’m going to teach you all about job searching. Give the seminar, and everyone gets a job. Now I need a control group. So I find a bunch of people who didn’t come to my seminar, and none of them get a job.
The people chose to show up. One explanation could be that they are super motivated people since they showed up at 7 am on a Saturday because they will do anything to get a job.
What would be more accurate is putting up more signs. 100 people show up. 50 get the seminar, and 50 don’t. If, in this case, the job seminar people got jobs, now there is more confidence since there was random assignment.
IV/ DV
IV is the group difference. Whatever variable has been altered in the group.
DV is the outcome.
Ex:
IV is job seminar or lack of job seminar. This is the one that the researcher has some control over. Then the outcome gets measured, and the researcher has no direct control over the outcome.
DV is whether the people got jobs or not.
Ex: Hot temperatures make you test worse. I get two classrooms and divide participants. One classroom is 95 degrees, and the other is 70 degrees, and see whose exam scores are higher.
IV is what I have control over. So the IV would be the room temperature.
DV is the exam scores.
Internal Validity vs Generalizability (e.g. kids/sesame street/ reading)
We were getting at this before when we took a look at naturalistic vs. laboratory.
Internal validity means that I’m really sure that whatever factor I messed with is the only thing I changed. Therefore, I can be pretty sure that is what caused the change.
Ex: The exam example: everything about the classrooms needs to be similar. Same comfort level of seats, randomly assign you to the two classrooms, same resources available. The only difference will be the temperature. That gives me a good sense of internal validity.
The more careful you are and the more you set up a labortory that controls for everything the less it looks like real life.
Generalizability: Is this going to apply in the real world?
Ex: Measured kids who watch Sesame street to see if it in fact helped with their reading skills.
I could bring the kids in once a week and they sit in this classroom. Everyone sits in the same desk and they aren’t allowed to have crayons or anything else going on. They sit and watch Sesame Street and I check if their reading gets better. This is highly internally valid.
Will this improve reading in the real world? No, because kids don’t sit at a desk staring at a screen with no distractions. How do I expect kids to use Sesame Street in the real world? Will there still be an effect in a real-world scenario?
The more it’s natural the less I’m sure. But the more careful I am the less I know it will work in the real world.
Ethics
In psycholoy we typically have something called an IRB (Institutional Review Board) that reviews the ethics of all studies. The basic prinicple is that the benefits need to outweight the costs. Keeping in mind that human pain and suffering is an incredibly high cost.
Every study that has been done has been reviewed by a team of people that get together to do nothing but talk about the ethics of it. They don’t care if the study is good or bad they just care the study is ethical.
In the past there have been horrically unethical studies.
Around the 1970s, we started to lay some of this out.
Informed consent is a big one.
You have to tell participants the benefits and costs might be and get a signature.
The only exception is being in a public place a researcher is allowed to take notes. No video can be taken.
There is a bunch more ethical guidelines.
Attrition
People drop out of studies.
Ex: Looking at things like cigarette smoking and age of death. Cross sectional studies at 60 and 30% of people are smokers. Then at age 90, 10% of people are smokers.
People stopped smoking or died.
This can skew data. This is why longitudial can be better than a cross sectional study.
Ex: Socioeconimic status on mental health development. Get a bunch of teens some are in lower economic status and some are in higher.
Over time people start dropping out of the study. The results will increasingly reflect who stayed in it.
Low income people are going to drop out because they have too many things and jobs to do.
I’m going to end up with a sample of very well educated, very handsomely paid. And I’m going to be drawing information from a group that is no longer representative of the population.