Enhancing Deep Knowledge Tracing via Diffusion Models

Enhancing Deep Knowledge Tracing via Diffusion Models for Personalized Adaptive Learning

Abstract

• This paper introduces generative AI models to enhance Deep Knowledge Tracing (DKT) for Personalized Adaptive Learning (PAL).
• DKT models students' evolving knowledge to predict their future performance and to provide personalized recommendations.
• Generative AI models, rooted in deep learning, generate synthetic data to address data scarcity.
• The study employs TabDDPM, a diffusion model, to generate synthetic educational records to augment training data for enhancing DKT.
• Experiments on ASSISTments datasets validate the proposed method's effectiveness.
• AI-generated data by TabDDPM significantly improves DKT performance, especially with small training data and large testing data.

I. Introduction

• Personalized Adaptive Learning (PAL) monitors individual student progress and tailors the learning path to their unique needs.
• Knowledge Tracing (KT) tracks the progression of students’ understanding to anticipate their upcoming performance.
• Deep Knowledge Tracing (DKT) leverages deep learning to understand temporal patterns in student interactions.
• DKT surpasses conventional KT models across benchmark datasets.
• Generative AI (GAI) models leverage deep learning to create synthetic data, addressing data scarcity challenges.
• Diffusion models have emerged as state-of-the-art GAI models, outperforming GANs in image synthesis.
• This paper introduces diffusion models to address data scarcity challenges in student learning records, aiming to enhance DKT performance for PAL.
• The approach utilizes TabDDPM, a diffusion model capable of generating synthetic educational records for any tabular dataset.
• The study validates the proposed method’s effectiveness through experiments on ASSISTments datasets.

Contributions:

1. Pioneering exploration of resolving data shortage issues on DKT through GAI models, using diffusion models to generate educational records.
2. Experimental results show a significant improvement in DKT performance, with a consistent increase in DKT performance as the amount of AI-generated training data increases.

II. Methodology

• The paper addresses data scarcity in DKT by generating synthetic data using diffusion models.

A. Deep Knowledge Tracing (DKT)

• DKT leverages Recurrent Neural Networks (RNNs) to capture temporal dynamics within a sequence of student interactions.
• RNN network equations for DKT:
  • $h<em>t = \tanh(W</em>{hx}x<em>t + W</em>{hh}h<em>{t-1} + b</em>h)$
  • $y<em>t = \sigma(W</em>{yh}h<em>t + b</em>y)$
  • Where $\tanh(\cdot)$, $\sigma(\cdot)$ are applied element-wise.
  • The model is parameterized by an input weight matrix $W<em>{hx}$, a recurrent weight matrix $W</em>{hh}$, an initial state $h<em>0$, and a readout weight matrix $W</em>{yh}$.
  • Biases for the latent and readout units are denoted by $b<em>h$ and $b</em>h$.
  • Inputs $(x<em>t)$ can be one-hot encodings or compressed representations of a student’s action, while the prediction $(y</em>t)$ is a vector representing the probability of correctly answering each exercise.
• DKT surpasses traditional KT models on various benchmark datasets.

B. TabDDPM

• TabDDPM is a streamlined DDPM variant tailored for tabular data, accommodating mixed data types.
• A tabular data sample is structured as $x = [x<em>{num}, x</em>{cat1}, …, x<em>{catC}]$, encompassing $x</em>{num}$ numerical features and $C$ categorical features denoted by $x_{cati}$.
• TabDDPM employs multinomial diffusion for categorical/binary features and Gaussian diffusion for numerical features.
• The reverse diffusion process uses a multi-layer neural network.
• Loss function:
  • $L<em>{TabDDPM}(t) = L</em>{simple}(t) + \frac{\sum<em>{i \leq C} L</em>i(t)}{C}$
  • $L_{simple}(t)$ is the mean-squared error for Gaussian diffusion.
  • $L_i(t)$ represents KL divergences for each multinomial diffusion.

C. Proposed Method

• The method involves randomly sampling data, using TabDDPM to generate synthetic education data, integrating synthetic data with original training data, training the DKT model, and evaluating the model using testing education data.

III. Experiment

A. Dataset

• ASSISTments datasets consist of longitudinal data from the ASSISTment platform, featuring math exercises from grade school.
• The study used ASSISTments2009, collected during the 2009-2010 school year, containing 525,535 interactions.
• The dataset was split into training (5%), validation (20%), and testing (75%) data.
• TabDDPM was applied to generate synthetic educational data, using skill id, user id, overlap_time, and ground truth correct.

B. Evaluation Metrics

• Evaluation metrics include Accuracy, Precision, Recall, and Area Under the Curve (AUC).
  • Accuracy is calculated as: $Accuracy = \frac{N<em>{correct}}{N</em>{total}}$
  • Precision is calculated as: $Precision = \frac{TP}{TP + FP}$
  • Recall is calculated as: $Recall = \frac{TP}{TP + FN}$
    • Where:
      • TP (True Positive) is the number of student answers that match the ground truth.
      • FP (False Positive) is the number of student answers that do not align with the ground truth but are incorrectly identified as correct.
      • FN (False Negative) is the number of instances where correct student answers are incorrectly classified as incorrect.
• AUC gauges the binary classifier’s efficacy in distinguishing between classes.

C. Experimental Results and Discussion

• The proposed model surpasses DKT in terms of accuracy, AUC, precision, and recall.
• TabDDPM-generated samples contribute significantly to enhancing AUC, recall, and accuracy, with a pronounced impact on reducing false negatives to increase the recall.
• Increasing the number of generated samples increases stability reflected in the standard deviation of accuracy, precision and recall.
• Increasing the quantity of generated data is correlated with enhanced performance.

IV. Related Work

1) Knowledge Tracing:

• KT monitors a student’s learning progress, representing and quantifying their knowledge state.
• KT can be categorized into traditional knowledge tracing (Bayesian Knowledge Tracing, Factor Analysis Models) and deep learning-based knowledge tracing.
• RNN-based techniques have significantly advanced the development of deep knowledge (DK) in KT.
• A persistent challenge in enhancing KT is the limited availability of education data.

2) Generative Models for Tabular Data Generation:

• Generative models for tabular data are explored due to the increasing demand for high-quality synthetic data.
• Tabular datasets, especially in education, are often limited in size compared to vision or NLP datasets.
• Synthetic datasets allow for public sharing without compromising anonymity.
• Models for tabular data generation include tabular VAEs, GAN-based approaches, and diffusion model-based approaches.

V. Conclusion and Future Work

• Diffusion models were utilized to improve deep knowledge tracing, with a specific focus on applying TabDDPM to generate synthetic education data.
• Future research includes extending the generation of synthetic education data and validating the proposed model in the context of personalized learning path recommendation.