133F Post Midterm 2 Week 10 Lecture 1

Student Evaluation

1) carpenter (2013)

2) Gilbert (2025) & Gilbert

carpenter (2013)

Appearances can be deceiving: instructor fluency increases perceptions of learning without increasing actual learning

Gilbert (2025)

Student evaluations of teaching do not reflect student learning: an observational study

Carpenter et al found that

A fluent instructor may increase perceptions of learning without increasing actual learning

Carpenter et al. found that the highest levels of perceived learning/ perceived performance were found among

students who’d listened to a fluent speaker

Gilbert and Gilbert studied student evaluations. what theoretical explanation did their study support?

grade bias theory

How should teacher quality be measured?

who should do it? → students and student experience, student performance on standardized tests, rate my professor, other people knowledgeable in the same field

on what basis should those individuals form a judgment?

how ARE teachers being evaluated?

• their level of education or training

• standardized student achievement tests SAT

• principal or coach ratings from classroom observations

• National Board Certification

• content knowledge tests

• pedagogical content knowledge tests

• parent evaluations

• student evaluations

Article #1 carpenter

appearances can be deceiving

lecture qualities

question the article takes on is the issue of INSTRUCTOR FLUENCY

• we want instructors to be fluent speakers to present information clearly in an organized way

• to be prepared/have preparedness

• ease of presentation

• expressiveness

if we know how fluent an instructor is, is that related to students actual learning?

• If you have a fluent instructor, the students should be able to understand the material more easily and be able to process it more easily.

interested in whether instructor fluency, having a well-organized instructor, can have an impact on perceived learning?

• How much you think you learned

also possible that fluency can have an impact on studying behavior

article 1 study 1 method

• n = 42 UGs from iowa state

• they were randomly assigned to 1 of 2 videos in this LAB STUDY

• same instructor, content, camera position, length

manipulated fluency by:

Video 1 fluent speaker → stood upright, spoke fluently without notes, maintained eye contact

Video 2 disfluent speaker → hunched over podium, read haltingly from notes, broke eye contact to flip through notes

• Ss watched video in a private room

• they were told to expect a memory test/prepare for a memory test

article 1 study 1 DVs (immediately after)

• Fluency (5 pt scale)

  • instructor fluency 1,2,3,4,5?

  • how ORGANIZED was the speaker in the video?

  • how PREPARED was the speaker in the video?

  • how KNOWLEDGEABLE was the speaker in the video?

• Effectiveness (5 pt scale)

  • instructor effectiveness 1,2,3,4,5?

  • please rate the overall effectiveness of the speaker in the video

• Estimation of Recall (0-100%)

  • how much of the video would you recall?

  • in about 10 minutes from now, how much of the information from the video do you think you’ll be able to recall?

• Perceived Learning (5 pt scale)

  • how well do you feel you’ve learned the information that was presented in the video?

• Interest (5 pt scale)

  • please rate your overall interest in the information that was presented in the video?

• Motivation to Learn (5 pt scale)

  • please rate your overall level of motivation to learn the information that was presented in the video?

article 1 study 1 after measuring for DV’s

• given a ten minute distraction task

• given 5 minutes for a free recall task having them recall all they could from the video

  • measure of actual learning

• before the DV’s were measuring perceived learning, this task attacks the actual learning

predicted vs actual learning

• there is a significant difference among the students who got the fluent speaker.

  • Their predicted level of learning far exceeded their actual learning.

• there is no significant difference among the students who got the disfluent speaker

  • Their predicted level of learning was pretty close to the actual level of learning.

• if you had the fluent speaker, you thought/predicted you were going to recall a whole lot more than if you had the disfluent speaker

  • but when it comes to actual learning, even if you have the fluent speaker, there is no significant difference between your actual performance in the fluent vs disfluent category

Bar A where the predicted is wayyy higher than actual for fluent is concerning

• it suggests that your metacognition, your awareness of what you know is off.

• that can affect subsequent behavior

• that level of overconfidence is a concern

article 1 study 1 more results

the fluent speaker was rated as:

• more organized

• more knowledgeable

• more prepared

• more effective

  • even though they weren’t more effective

ratings:

• perceived levels of learning: fluent instructor > disfluent instructor

• interest levels: fluent instructor = disfluent instructor

• motivation levels: fluent instructor = disfluent instructor

article 1 study 1 conclusion

• instructor fluency you can think of as biasing the Ss’ judgments of learning

• If you have a fluent instructor, there creates a bias, and you end up thinking that you've learned more than you have.

open question from Study 1

• when Ss think they are learning less, do they compensate with studying more?

article 1 study 2 methods

• n = 70 UG

• same basic design as study 1

  • THIS TIME→ they allowed Ss to study after watching the video (test based script; no time limit)

  • could study the transcript of the video for as long as they wanted

  • DV = study time

• then given same process of distractor task, rating, and test

article 1 study 2 results

• actual learning was the same across conditions (like study 1)

• the fluent instructor was rated as:

  • more organized

  • more knowledgeable

  • more prepared

  • more effective (like study #1)

• perceived levels of learning: fluent instructor > disfluent instructor (like study 1)

• interest levels: fluent instructor = disfluent instructor (like study 1)

• motivation levels: fluent instructor > disfluent instructor (different from study 1)

• study time was the same across all conditions

a more fluent instructor may increase perceptions of learning without increasing actual learning or affecting studying decisions

instructor fluency:

• was not associated with actual learning (did not make you learn any more)

• did have an impact on perceived learning (having a very fluent instructor made you feel like you did learn more even when you did not)

• did not influence studying behavior (did not cause students to study more when they felt like they werent learning)

why should we care?

• a fluent instructor can lead you to be overconfident

  • that can lead you to think you might not need to study, or it might lead you to abort your studying prematurely

• if you want to learn, it helps to have a clear awareness of what you know

  • really has to do with metacognition

  • if you want to learn, it helps to have a clear awareness of what you know

  • what you’ve already learned, what you still need to study, but you still need to act on it

  • metacognition is important and your metacognition can be biased by your perception of instructor fluency

article 2 gilbert 2025

student evaluations dont reflect student learning

student evaluations of teaching SET

• they are meant to be evaluations of teaching effectiveness

  • they measure how well you learned and how how well the course offered opportunities for you to learn the material.

• used to inform faculty employment reviews (promotion, tenure)

  • part of the 3 year teacher review process

article 2 study 1 assumption

• students reported perceptions of teaching quality accurately measure actual teaching quality

article 2 study 1 evidence of bias

• there is evidence that student evaluations are biased.

• by gender, physical attraction, race, ethnicity, age, culture, prior interest in topic, perception of leniency

• high grades or expectations of high grades

  • two theories:

  • teacher effectiveness theory: higher evaluations serve as evidence of more learning (ex good teaching)

    • if you expect a high grade it's a reflection of really good teaching & that you had a good teacher.

  • Grade based theories: higher grades drive favorable evaluations

    • it could also be that high grade or the expectation of a high grade, or the belief that your instructor is an easy grader leads to more favorable evaluations.

article 2 study 1 goal

evaluate the relationship between student evaluations and:

• course grades

• independent measures of learning

article 2 study 1 method

• n=160 courses between fall 2018 and summer 2022

  • 12 cohorts of students

• all Ss took all courses in the same sequence

• same evaluation form was used across all courses

• SET was conducted online between 3 weeks before the end of the course and 2 weeks after the course ended

• completion was optional and anonymous

article 2 study 1 measures

• SET scores

• Course Grades, looking at:

  • individual student scores

  • % of students in the A range & % of students on the F range

• Veterinary Educational Assessment

  • external exam administered by National Board of Medical Examiners

  • taken at the end during 5th semester

  • 240 items

  • results correlated with Veterinary License Exam

article 2 study 1 results

• Mean course evaluations were significantly and positively correlated with

  • median grade and

  • with the percentage of grades that fell into the A range

• mean course evaluations were significantly negatively correlated with

  • corresponding sectional VEA scores (that external measure)

  • percentage of grades in the A-range

• positive correlation between grade and course evaluation

• negative correlation between VEA score and course evaluation

  • That's problematic, suggesting this kind of relationship between evaluations and grades

    • that the people who were getting high grades gave high course evaluations, but those people were doing poorly on the exam

why should we care?

• evaluations did not reflect actual learning

• evaluations might cause instructors to inflate grades

• SET should be used with caution when informing consequential decisions

• if we want to measure effectiveness of teaching, we might need a different measure