Reinforcement Learning: Foundations, Probability, and Applications

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/29

flashcard set

Earn XP

Description and Tags

Comprehensive vocabulary flashcards covering the core concepts of Reinforcement Learning Unit I, including intelligent agents, sequential decision-making, exploration vs. exploitation, mathematical foundations of probability, linear algebra, and real-world RL applications.

30 Terms

1
New cards

Reinforcement Learning (RL)

A machine learning paradigm that deals with sequential decision-making problems where there is no target value (correct answer), but only scalar positive or negative rewards that may be delayed in time.

2
New cards

Agent (in RL)

The decision-maker and active learner in the system that perceives the current state of the environment and chooses which action to execute.

3
New cards

Environment (in RL)

Everything outside the agent; the world that the agent interacts with, reacts to, receives rewards from, and learns about.

4
New cards

Reward (RtR_t)

An instantaneous scalar numerical feedback signal returned by the environment to the agent at time step tt in direct response to the agent's action.

5
New cards

State (StS_t)

A concrete numerical or tensor-based representation of the environment's situation at a specific time step tt.

6
New cards

Return (GtG_t)

The total cumulative sum of rewards that an agent expects to accumulate from the current time step tt into the future, which the agent seeks to maximize.

7
New cards

Policy (π\pi)

The agent's internal behavior function or decision rule that maps states to actions, defined formally as the conditional probability distribution π(a∣s)=P(At=a∣St=s)\pi(a \mid s) = P(A_t = a \mid S_t = s).

8
New cards

Value Function

The agent's prediction of the long-term desirability of a state (or state-action pair), representing the expected cumulative future reward starting from that state.

9
New cards

Environment Model

The agent's internal representation or simulation of the environment's transition dynamics, allowing it to predict future states and rewards for planning before executing actions.

10
New cards

Actor-Critic Model

A reinforcement learning architecture that combines both a policy (the actor) to choose actions and a value function (the critic) to evaluate the chosen actions.

11
New cards

Perception-Decision-Action Loop

The repeating cyclic interaction in RL where the agent observes state StS_t (perception), evaluates StS_t to select action AtA_t (decision), and executes AtA_t to transition the environment to St+1S_{t+1} and receive reward Rt+1R_{t+1} (action).

12
New cards

Episodic Task

A task that naturally terminates upon reaching a definitive terminal state at a finite horizon, where the discount factor γ\gamma is optional (0≤γ≤10 \le \gamma \le 1) and the return sum is naturally finite.

13
New cards

Continuous Task

A task that runs indefinitely without a natural endpoint (T=∞T = \infty), requiring a mandatory discount factor (0≤γ<10 \le \gamma < 1) to prevent the infinite return sum from diverging to infinity.

14
New cards

Discount Factor (γ\gamma)

A scalar factor (0≤γ<10 \le \gamma < 1) used to weight immediate rewards more heavily than distant future rewards and ensure convergence of infinite-horizon returns.

15
New cards

Multi-Armed Bandit (MAB)

A simplified non-sequential RL problem consisting of a single stationary state where the agent's actions do not affect future states, designed primarily to study the exploration versus exploitation dilemma.

16
New cards

Exploitation (Greedy Choice)

The selection of the action with the highest currently estimated value in order to maximize immediate reward based on current knowledge.

17
New cards

Exploration

The selection of non-greedy actions to gather additional information about alternative actions whose true payouts might exceed current estimates.

18
New cards

Credit Assignment Problem

The challenge of identifying which specific actions in a long sequence were responsible for an eventual delayed reward received at the end of a trajectory.

19
New cards

Aleatoric Uncertainty

Inherent, irreducible randomness built into the physical universe or process (such as rolling a fair die).

20
New cards

Epistemic Uncertainty

Knowledge-based uncertainty arising from the agent's ignorance or lack of experience, which can be systematically reduced through exploration.

21
New cards

Independent Events

Two events AA and BB such that knowing the outcome of one does not affect the probability of the other, satisfying P(A∣B)=P(A)P(A \mid B) = P(A) and P(A∩B)=P(A)×P(B)P(A \cap B) = P(A) \times P(B).

22
New cards

Mutually Exclusive Events

Disjoint events that cannot occur at the same time, characterized mathematically by an empty intersection (P(A∩B)=0P(A \cap B) = 0).

23
New cards

Bayes' Theorem

A foundational theorem of conditional probability for updating beliefs based on evidence, given by P(A∣B)=P(B∣A)P(A)P(B)P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}.

24
New cards

Permutation

An ordered sequence of kk objects selected from a set of nn distinct objects, calculated as Pk,n=n!(n−k)!P_{k,n} = \frac{n!}{(n - k)!}.

25
New cards

Combination

An unordered subset of kk objects chosen from a set of nn distinct objects, expressed as (nk)=n!k!(n−k)!\binom{n}{k} = \frac{n!}{k!(n - k)!}.

26
New cards

Marginalization

The operation of eliminating an unwanted variable from a joint probability distribution by summing or integrating across all its values, such as computing p(s′∣s,a)=∑r∈Rp(s′,r∣s,a)p(s' \mid s, a) = \sum_{r \in \mathcal{R}} p(s', r \mid s, a).

27
New cards

Kelly Criterion

A mathematical formula (f∗=p−qbf^* = p - \frac{q}{b}) created by John L. Kelly Jr. in 1956 to determine the optimal fraction of a bankroll to wager to maximize long-term compound wealth growth while avoiding ruin.

28
New cards

Sim2Real Gap

The reality discrepancy that occurs when a robotics RL policy trained in a physics simulator (such as MuJoCo or PyBullet) fails or underperforms when deployed on a real physical robot.

29
New cards

Adjugate Matrix

The transpose of the cofactor matrix, which for a 2×22 \times 2 matrix A=(abcd)A = \begin{pmatrix} a & b \\ c & d \end{pmatrix} equals adj⁡(A)=(d−b−ca)\operatorname{adj}(A) = \begin{pmatrix} d & -b \\ -c & a \end{pmatrix}, satisfying A−1=1det⁡(A)adj⁡(A)A^{-1} = \frac{1}{\det(A)}\operatorname{adj}(A).

30
New cards

Markov Chain (Historical Origin)

A mathematical framework introduced by Andrey Andreyevich Markov in 1913 through the empirical analysis of vowel and consonant transition probabilities across 20,000 letters of Alexander Pushkin's Eugene Onegin.