1/44
Looks like no tags are added yet.
LoRA path
background, mechanism, training, outcome
LoRA background
weights, gradients, training memory
LoRA mechanism
two small matrices adapt a model
LoRA training cycle
forward pass, measure error, backward pass, adjust weights
What happens to prediction when we measure loss
penalize
In LoRA what happens when we find directions
calculate how each weight affects the penalty
LoRA first steps
freeze the original weights, store them for small LoRA adapters, save adapters plus base-model reference
What do we story in training memory
weights, gradients, master weights, adam optimizer history
adam optimizer history
actual part to update weights based on gradients
why does adam keep extra history
an optimizer turns gradients into changes to the weights
average gradient
the recent directions of the gradients, smoothed over steps
average squared gradient
recent size of the gradients
what are optimizer states used for
adam uses them to choose weight updates
optimizer states
average gradient, average squared gradient
activations
intermediate values saved while predicting
FP16
16-bit floating point 2 bytes
what is stored in FP16
smaller working copy of the weights
FP32
32-bit floating point 4 bytes
what is FP32 used for
more precision for updates and history
In LoRA’s W = W₀ + sBA what is W0 (mxn) and what is A and B
pretrained frozen matrix, trainable adapter matrices
rank
controls capacity and parameter count
For one mxn matrix, LoRA =
r(m+n)
output scale s
a/r
LoRA forward pass
h = W0x + sB(Ax)
W0x in LoRA forward pass
pretrained model
A in LoRA forward pass
compresses n features to r
B in LoRA forward pass
maps r features to m
Initialization in LoRA forward pass
begin with original model
merged weight matrix at inference
h = Wmerged x
Advantages of LoRA
fewer trainable parameters, task-specific adapters, adapters have one compatible base, can merge for inference
Disadvantages of LoRA
base model in memory, rank and target limit update, lose some info
QLoRA
quantizing the precision of the network parameters
quantization
split rang into buckets
we can quantize F32 into
INT8
Three key ingredients of QLoRA
NF4, double quantization, paged optimize
What is 4-bit normalfloat used for before tuning
quantized base weights
What is 4-bit normalfloat used for during each prediction
reconstructed weights in BF16
double quantization q = round(c * w)
compressing the scales to reduce memory
paged optimize
managed memory peaks
paged optimizers borrow space from
CPU RAM
page
small block of memory
activation checkpointing
discard some activations and recompute them later
paged optimizer
move eligible optimizer states between GPU memory and CPU RAM
QLoRA steps
store frozen weight in NF4, double quantization stores block scales, forward pass, backward pass, paged optimizer states provide extra room
QLoRA equation: h = Ŵ x + (α / r) B(Ax) what is Ŵ
dequantize(qW, reconstructed scales)