CSE 534 quiz 3

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/44

encourage image

There's no tags or description

Looks like no tags are added yet.

45 Terms

1
New cards

LoRA path

background, mechanism, training, outcome

2
New cards

LoRA background

weights, gradients, training memory

3
New cards

LoRA mechanism

two small matrices adapt a model

4
New cards

LoRA training cycle

forward pass, measure error, backward pass, adjust weights

5
New cards

What happens to prediction when we measure loss

penalize

6
New cards

In LoRA what happens when we find directions

calculate how each weight affects the penalty

7
New cards

LoRA first steps

freeze the original weights, store them for small LoRA adapters, save adapters plus base-model reference

8
New cards

What do we story in training memory

weights, gradients, master weights, adam optimizer history

9
New cards

adam optimizer history

actual part to update weights based on gradients

10
New cards

why does adam keep extra history

an optimizer turns gradients into changes to the weights

11
New cards

average gradient

the recent directions of the gradients, smoothed over steps

12
New cards

average squared gradient

recent size of the gradients

13
New cards

what are optimizer states used for

adam uses them to choose weight updates

14
New cards

optimizer states

average gradient, average squared gradient

15
New cards

activations

intermediate values saved while predicting

16
New cards

FP16

16-bit floating point 2 bytes

17
New cards

what is stored in FP16

smaller working copy of the weights

18
New cards

FP32

32-bit floating point 4 bytes

19
New cards

what is FP32 used for

more precision for updates and history

20
New cards

In LoRA’s W = W₀ + sBA what is W0 (mxn) and what is A and B

pretrained frozen matrix, trainable adapter matrices

21
New cards

rank

controls capacity and parameter count

22
New cards

For one mxn matrix, LoRA =

r(m+n)

23
New cards

output scale s

a/r

24
New cards

LoRA forward pass

h = W0x + sB(Ax)

25
New cards

W0x in LoRA forward pass

pretrained model

26
New cards

A in LoRA forward pass

compresses n features to r

27
New cards

B in LoRA forward pass

maps r features to m

28
New cards

Initialization in LoRA forward pass

begin with original model

29
New cards

merged weight matrix at inference

h = Wmerged x

30
New cards

Advantages of LoRA

fewer trainable parameters, task-specific adapters, adapters have one compatible base, can merge for inference

31
New cards

Disadvantages of LoRA

base model in memory, rank and target limit update, lose some info

32
New cards

QLoRA

quantizing the precision of the network parameters

33
New cards

quantization

split rang into buckets

34
New cards

we can quantize F32 into

INT8

35
New cards

Three key ingredients of QLoRA

NF4, double quantization, paged optimize

36
New cards

What is 4-bit normalfloat used for before tuning

quantized base weights

37
New cards

What is 4-bit normalfloat used for during each prediction

reconstructed weights in BF16

38
New cards

double quantization q = round(c * w)

compressing the scales to reduce memory

39
New cards

paged optimize

managed memory peaks

40
New cards

paged optimizers borrow space from

CPU RAM

41
New cards

page

small block of memory

42
New cards

activation checkpointing

discard some activations and recompute them later

43
New cards

paged optimizer

move eligible optimizer states between GPU memory and CPU RAM

44
New cards

QLoRA steps

store frozen weight in NF4, double quantization stores block scales, forward pass, backward pass, paged optimizer states provide extra room

45
New cards

QLoRA equation: h = Ŵ x + (α / r) B(Ax) what is Ŵ

dequantize(qW, reconstructed scales)