1/29
Comprehensive vocabulary flashcards covering the core concepts of Reinforcement Learning Unit I, including intelligent agents, sequential decision-making, exploration vs. exploitation, mathematical foundations of probability, linear algebra, and real-world RL applications.
Reinforcement Learning (RL)
A machine learning paradigm that deals with sequential decision-making problems where there is no target value (correct answer), but only scalar positive or negative rewards that may be delayed in time.
Agent (in RL)
The decision-maker and active learner in the system that perceives the current state of the environment and chooses which action to execute.
Environment (in RL)
Everything outside the agent; the world that the agent interacts with, reacts to, receives rewards from, and learns about.
Reward (Rt)
An instantaneous scalar numerical feedback signal returned by the environment to the agent at time step t in direct response to the agent's action.
State (St)
A concrete numerical or tensor-based representation of the environment's situation at a specific time step t.
Return (Gt)
The total cumulative sum of rewards that an agent expects to accumulate from the current time step t into the future, which the agent seeks to maximize.
Policy (π)
The agent's internal behavior function or decision rule that maps states to actions, defined formally as the conditional probability distribution π(a∣s)=P(At=a∣St=s).
Value Function
The agent's prediction of the long-term desirability of a state (or state-action pair), representing the expected cumulative future reward starting from that state.
Environment Model
The agent's internal representation or simulation of the environment's transition dynamics, allowing it to predict future states and rewards for planning before executing actions.
Actor-Critic Model
A reinforcement learning architecture that combines both a policy (the actor) to choose actions and a value function (the critic) to evaluate the chosen actions.
Perception-Decision-Action Loop
The repeating cyclic interaction in RL where the agent observes state St (perception), evaluates St to select action At (decision), and executes At to transition the environment to St+1 and receive reward Rt+1 (action).
Episodic Task
A task that naturally terminates upon reaching a definitive terminal state at a finite horizon, where the discount factor γ is optional (0≤γ≤1) and the return sum is naturally finite.
Continuous Task
A task that runs indefinitely without a natural endpoint (T=∞), requiring a mandatory discount factor (0≤γ<1) to prevent the infinite return sum from diverging to infinity.
Discount Factor (γ)
A scalar factor (0≤γ<1) used to weight immediate rewards more heavily than distant future rewards and ensure convergence of infinite-horizon returns.
Multi-Armed Bandit (MAB)
A simplified non-sequential RL problem consisting of a single stationary state where the agent's actions do not affect future states, designed primarily to study the exploration versus exploitation dilemma.
Exploitation (Greedy Choice)
The selection of the action with the highest currently estimated value in order to maximize immediate reward based on current knowledge.
Exploration
The selection of non-greedy actions to gather additional information about alternative actions whose true payouts might exceed current estimates.
Credit Assignment Problem
The challenge of identifying which specific actions in a long sequence were responsible for an eventual delayed reward received at the end of a trajectory.
Aleatoric Uncertainty
Inherent, irreducible randomness built into the physical universe or process (such as rolling a fair die).
Epistemic Uncertainty
Knowledge-based uncertainty arising from the agent's ignorance or lack of experience, which can be systematically reduced through exploration.
Independent Events
Two events A and B such that knowing the outcome of one does not affect the probability of the other, satisfying P(A∣B)=P(A) and P(A∩B)=P(A)×P(B).
Mutually Exclusive Events
Disjoint events that cannot occur at the same time, characterized mathematically by an empty intersection (P(A∩B)=0).
Bayes' Theorem
A foundational theorem of conditional probability for updating beliefs based on evidence, given by P(A∣B)=P(B)P(B∣A)P(A).
Permutation
An ordered sequence of k objects selected from a set of n distinct objects, calculated as Pk,n=(n−k)!n!.
Combination
An unordered subset of k objects chosen from a set of n distinct objects, expressed as (kn)=k!(n−k)!n!.
Marginalization
The operation of eliminating an unwanted variable from a joint probability distribution by summing or integrating across all its values, such as computing p(s′∣s,a)=∑r∈Rp(s′,r∣s,a).
Kelly Criterion
A mathematical formula (f∗=p−bq) created by John L. Kelly Jr. in 1956 to determine the optimal fraction of a bankroll to wager to maximize long-term compound wealth growth while avoiding ruin.
Sim2Real Gap
The reality discrepancy that occurs when a robotics RL policy trained in a physics simulator (such as MuJoCo or PyBullet) fails or underperforms when deployed on a real physical robot.
Adjugate Matrix
The transpose of the cofactor matrix, which for a 2×2 matrix A=(acbd) equals adj(A)=(d−c−ba), satisfying A−1=det(A)1adj(A).
Markov Chain (Historical Origin)
A mathematical framework introduced by Andrey Andreyevich Markov in 1913 through the empirical analysis of vowel and consonant transition probabilities across 20,000 letters of Alexander Pushkin's Eugene Onegin.