Replace a dense update with two factors
Let a frozen linear layer use W₀ with shape d_out × d_in. A low-rank adapter represents its change as ΔW = (α/r) B A, where A has shape r × d_in and B has shape d_out × r. The layer computes W₀ x + (α/r) B(Ax). The base weights stay frozen while the factors are trained. The factorization uses r(d_in + d_out) trainable parameters. A separate full dense update would use d_in d_out parameters. The ratio is favorable when r is small relative to both dimensions. The rank of B A cannot exceed r, so the smaller parameter count also restricts the updates that can be represented. For a concrete shape check, choose d_in = 4, d_out = 3 and r = 2. A maps a four-dimensional input to two dimensions; B maps those two dimensions to three. Their product has the same 3 × 4 shape as W₀. This notebook describes the core linear-algebra construction. Actual adapter configurations choose which layers to target and may include other trained parameters.