AI – Kurzkarten RL, CV, NLP

Agent

The learner that takes actions.

Environment

Everything the agent interacts with; returns reward + next state.

State (s)

Current situation of the agent.

Action (a)

What the agent does in a state.

Reward (r)

Immediate feedback after an action.

Policy (π)

Rule: which action in which state. π(s,a) = p(a | s)

Value V^π(s)

Long-term expected reward from state s when following π.

Reward vs. Value

Reward = immediate. Value = long-term (sum of future rewards).

Return (undiscounted)

R_t = r_{t+1} + r_{t+2} + … + r_T

Discount factor γ

0 ≤ γ ≤ 1. Near 1 = far-sighted, near 0 = short-sighted.

Discounted value formula

V^π(s) = E_π[ Σ γ^k · r_{t+k+1} | s_t = s ]

Markov property

Next state depends only on current state + action, not the past.

MDP components

States S, Actions A, Rewards R, Transition probabilities P

Model-based RL

Agent learns a model of the environment.

Model-free RL

Agent learns directly from experience, no model. Example: Q-learning.

Q-value Q(s,a)

Value of taking action a in state s.

Q-learning update (course)

Q(s,a) = r + γ · max Q(s',a')

Q-table start value

All values = 0.

Learning rate α

How much new info overwrites the old Q-value (0 = no learning, 1 = full replace).

Exploration

Try random actions to learn more.

Exploitation

Pick the best known action (max Q-value).

Q-learning steps

1) choose action 2) perform it 3) observe reward + update Q-table

Pinhole camera: 3 coordinate systems

World, Camera, Image

Pinhole camera: 2 steps

1) World → Camera (extrinsic) 2) Camera → Image (intrinsic)

Extrinsic parameters

Camera position + orientation: Rotation R and Translation t.

Intrinsic parameters

Inside the camera: focal length f, principal point (c_x, c_y), skew.

Tokenization

Split text into words or tokens.

Stemming

Cut word to its stem by rules (running → run).

Lemmatization

Reduce word to dictionary form (better → good).

Stop words

Very common words removed (the, is, and).

Bag of Words

Text as word counts, word order ignored.

TF-IDF

Word weight = frequency in document × rarity across documents.

Word embedding

Word as a vector; similar meaning = close vectors (e.g. word2vec).

POS tagging

Label word type (noun, verb, …).

NER

Named Entity Recognition: find names, places, organisations.

RNN

Reads sequence step by step; weak on long dependencies.

Transformer

Uses self-attention, processes all tokens in parallel.