Chapter 1 & 2: Introduction and Assessment Submission
Introduction and Welcome
Pete Edwards and Jenny welcome the students and introduce Nelly Constantinova, a Student Partnerships Advisor from the University of Leeds. Nelly's team focuses on enhancing teaching, learning, and assessment through partnerships with students. Jenny, a learning designer in the digital education service, shares her extensive teaching experience. The session aims to gather student feedback on the capstone project rubric, improve online assessment communication, and initiate ongoing consultation with students on assessment criteria.
Objectives of the Session
The primary goals are to:
Reflect on the challenges students face in understanding the capstone project rubric.
\
Actively consult with students to critique and refine assessment criteria.
Valuing Student Contributions
The input from students is considered invaluable, particularly from those involved in teaching and delivering data science modules. The Digital Education Service (DES) emphasizes the importance of understanding how students engage with course materials. Student insights are viewed as unique, fresh, relevant, and crucial for improving course design and engagement for future cohorts.
Guidelines for Participation
To maximize the benefit of the session, participants are encouraged to:
Be present and engaged.
Maintain a positive and solution-focused attitude.
Develop and build upon the ideas of others.
Use the chat function to share comments and ideas.
Remember that all ideas are welcome and valuable.
Student Introductions and Assessment Experiences
Nelly invites the students to introduce themselves, sharing their backgrounds, motivations for joining the program, and experiences with assessments. Key questions include:
How confident did you feel when submitting assessments?
Were there misalignments between expected and actual marks?
Were assessment criteria clearly communicated?
Flo, with a background in physics and astronomy, discusses her experience in the pharmaceutical and healthcare industry as a data scientist and analyst and enrolled in the master's program to formalize on-the-job learning. She favored the machine learning module assessment due to its flexibility and freedom in choosing research questions and datasets. Flo notes a common issue across educational institutions in the UK: the perceived unobtainability of the highest grades, even with exceptional work. She mentions receiving feedback which seemed perfunctory, such as a minor suggestion about adding percentages to a pie chart, which didn't seem commensurate with the mark received. This raises a point about the compression of grades and the reluctance to fully utilize the mark scheme's range.
Discussion on Grade Compression
The conversation expands to the issue of grade compression in UK universities, where achieving marks above 90% is rare, regardless of the effort invested in a project comparing it to movie ratings where the lower end of the scale is rarely used.
Student Experiences with Assessment
Mohammed Amf, a data scientist from Egypt working for the Bank of London in the Middle East, aims to deepen his data science skills through the program. He enjoyed the first module, supported data analytics, due to its freshness and excitement. While generally achieving good marks (above 70%), he experiences a lack of predictability in assessment outcomes. Sometimes, high confidence leads to lower marks, and vice versa. He recalls an instance in the first module where failing to round results to the specified decimal place resulted in a significant mark deduction, despite achieving correct results which highlights the importance of following instructions precisely. He notes that time constraints during exams added to the pressure.
The Assessment Rubric: Structure and Purpose
Pete explains the structure of the assessment rubric, which includes:
Assessment criteria (what is being assessed).
Classifications (grade boundaries).
Weighted distribution of marks for each criterion.
The aim is to provide a clear, progressively developing picture of how to enhance work to achieve higher grades. Pete expresses concern about students' confidence in decoding the rubric's quality standards.
How Students Use Assessment Rubrics
Pete inquires about how students use assessment rubrics during their learning process, whether to support learning throughout a module or only when approaching an assessment.
Mohammed uses the rubric as a retrospective checklist after completing the assessment to ensure all points are covered.
Flo reviews the learning objectives at the beginning of each unit as a preliminary guide. For untimed project modules, she completes the project first and then cross-compares it with the rubric from a third-party perspective to assess whether the work meets the criteria. She notes doing this retrospectively.
Confidence and Rubric Use and Time Constraints
In timed assessments, Flo prefers to study comprehensively due to a previous experience where unexpected questions led to a lower grade. This approach aims to prevent surprises, regardless of the rubric's stated coverage. Mohammed focuses on achieving the highest mark, using the rubric as a final check, but not during the initial learning phase.
Improving the Usefulness of Rubrics
Suggestions for improvements include:
Adjusting the level of detail.
Improving presentation.
Providing examples.
Using clearer wording.
Creating a checklist.
Flo suggests that the ease of meeting rubric criteria may depend on the assigned topic. She also suggests example works.
Examples and Project Scope
Jenny confirms that students will have a choice between four broad project briefs, each with a dataset to supplement. Flo expresses concern that some topics might be easier to align with the rubric than others. Jenny responds that the team is working on an example to illustrate this further. She adds the the focus is to think about how one approaches the problem, brings together multiple perspectives, and synthesizes them, versus PhD level depth. Flo and Mohammed both welcome examples.
Examples and Length
Flo mentions that many students would appreciate guidance on the expected length of assignments, even in open-ended projects and the lack of an upper limit can be stressful for some. Jenny mentions this is related to later questions.
Granularity
Mohammed suggests smaller, unit-specific rubrics in addition to the overall module rubric. He believes advance knowledge could lead to missing algorithms taught later, but also that unit-specific versions might help guide them through the units.
Example Grading Clarity Required
Flo says that when it comes to examples, she would like some examples that are not great and some that are great in what each earned, and to know what was deficient.
Motivations and Motivations for project
Jenny motivates the project by saying she felt that as a marker, it was difficult to give key feedback to students. She wanted to get down to granular details to say what earns a specific mark.
Ranking Assessment Criteria
Students are asked to rank the importance of assessment criteria.
Flo considers the "development of the research question" as the most important, as everything else falls from it. She deems "professional integrity" and ethical risks as the least important, as some topics may not have subject level data. Jenny qualifies this: the big ideas have national significance. She's looking for whether interpretations introduce some ethical aspects.
Mohammed considers the "presentation of posters" and "development of research questions" as the most important. The least important is "application of Crisp-DM framework" because there are many, and firms change and modify Crisp-DM.
Clarity and Language
The discussion shifts to specific elements of language within the rubric, focusing on terms that students may find difficult to interpret. One area of focus is with using "clear purpose and simple question."
Flo recommends providing examples to illustrate what constitutes a "simple question" rather than seeking synonyms.
Specific Language and Nuance
The discussion shifts to the phrase "trivial solution" to have a trivial solution.
Flo says that coming from mathematics, it could mean x = 0.
Research Question Definition
The discussion moves to the phrase "research question is specific and does not lend itself to a straightforward yes/no answer".
Flo says this runs into a contradiction, because that a simple research question usually has a yes no answer. So examples will help here.
Specificity Revisited
Flo feels a specific research question can have a yes or no answer. This can require either a lot of interpretation versus the trivial yes/no answer, but not interpretation.
Wording Suggestions
The discussion moves to technical language where it says "contains few errors and accurately uses technical language to justify the analysis and/or conclusions".
Mohammed responds that the phrase about containing few errors and yet being accurate is internally inconsistent.
The next thing to be precise with technical language.
Flo responds that as somebody who publishes journals, it makes a lot of sense. Be precise whether associations are causal versus correlational.
Additional Feedback
(this section includes elements from Chapters 1 and 2)
The discussion wraps up with Mohammed wanting to know how to provide integrity and ethics. The lesson and rubric do not show how to cover them all. Should cover one part of ethics or all things in the unit? It wasn't too clear.
Nelly reiterates that those who add their feedback after the teams meeting and continue the discussion in chat and or do some work.
Pete reiterates that the things that will be done with feedback for all participants: gather in responses in questionnaires and considering what's been said, review the rubric, highlight where we've taken your input into account because this is you know, we want to demonstrate that your views are valuable to us and that we've used them, We've taken your opinions into account. You know, it's not just tick box ex tick tick box exercise. It's it's a valuable, valuable exercise to us. And, hopefully, it'll lead to continued collaboration with with students.
Application of Crisp-DM - Examples and Discussion
Flo asks whether the team wishes to see them explicitly say what part of crisp-dm they're in. She thinks she remembers a part of the essays where she had to hit the elements, but not the header.
Jenny wants a system appropriate way of approaching data science projects. A way it can reasonably be used in another context. A following of logical sequence of having an ability to look back on itself. It will come out in the wash in the poster and has an beginning, middle and end.
Flo asked if the format should come up as the usual academic conference poster (introduction, methods, results, discussion)? Jenny said it should but you've got extra recording to say your narrative. It can base level be, here's beginning, my planning, modeling, conclusion.
Flo thinks the example poster should have comment boxes so they can guide people through the various elements.