SVD

Any m×n matrix A can be decomposed as A = U·Σ·Vᵀ, where U is an m×m orthogonal matrix, Σ is an m×n diagonal matrix, and Vᵀ is the transpose of an n×n orthogonal matrix.

PCA is equivalent to performing eigenvalue decomposition on the data covariance matrix; if SVD is applied to the centered data matrix X to obtain X=UΣVᵀ, then the right singular vectors V are the principal component directions of PCA, and the squared singular values divided by the number of samples are the variances corresponding to each principal component.

Performing SVD on the projection matrix W yields a text subspace corresponding to 512 non-zero singular values and null-space directions corresponding to 256 zero singular values.

PRSR's low-rank subspace estimation applies SVD or PCA to the matrix formed by the residual vectors of parallel sentence pairs, takes the top k singular vectors to span the subspace, and projects the final-layer token hidden states onto the orthogonal complement of that subspace.

Applying singular value decomposition (SVD) to the key and value weight matrices of a pretrained model can factorize them into a dimensionality-reduction projection (encoder) and a dimensionality-expansion projection (decoder), converting standard attention into a low-rank cache structure similar to multi-head latent attention (MLA).

Low-rank decomposition is typically implemented using Singular Value Decomposition (SVD), reducing the parameter count from m×n to r×(m+n).

AttSVD is a training-free KV cache compression scheme that performs low-rank compression of the KV cache along the feature axis via attention-guided SVD.

Muon applies the matrix sign function to the SGD momentum gradient of weight matrices—performing SVD on the matrix and setting all singular values to 1—which is equivalent to finding the orthogonal matrix closest to the gradient matrix.

The mathematical foundation of low-rank approximation comes from singular value decomposition (SVD) in linear algebra, which can approximate a matrix with a small effective rank using the product of two low-dimensional matrices U·V^T.

Early Latent Semantic Analysis (LSA) captured semantic relationships between words by performing SVD on a word-document matrix.

SVD decomposes an arbitrary m×n matrix A into the form A=UΣVᵀ, where U and V are orthogonal matrices and Σ is a diagonal matrix, and the solution to the homogeneous equation Ah=0 is the last column of V.

The most classical method for solving the homography matrix is the DLT (Direct Linear Transform) algorithm: the perspective equations are transformed into a homogeneous linear system Ah=0, and the right singular vector corresponding to the smallest singular value is obtained via SVD (Singular Value Decomposition); when more than 4 point pairs are available, a least-squares optimal solution is provided.

Σ对角线上的奇异值按从大到小排列

SVD（奇异值分解）对任意形状的矩阵都适用，不要求方阵，且对任何矩阵都必然存在

SVD在协同过滤推荐系统中可将用户-物品评分矩阵映射到低维潜在因子空间，从而预测用户对未评分物品的偏好，其变体包括SVD++、时序SVD等

SVD的思想连接着PCA、LSA、矩阵补全、低秩近似等一系列机器学习核心技术，深度学习中的低秩适配（LoRA）是一种高效微调大模型的技术

潜在语义分析（LSA）诞生于1988年，通过对词-文档矩阵（通常经TF-IDF加权）进行SVD并保留前k个奇异值，将词和文档映射到k维语义空间

SVD中Σ对角线上的奇异值按从大到小排列，最大的奇异值对应数据中最重要的变化方向，较小的奇异值往往对应噪声或次要信息

对一张1000×1000的灰度图像保留前50个奇异值只需存储约10万个数值（50×1000+50+50×1000），压缩率达到约10:1

Eckart-Young-Mirsky定理证明，在所有秩为k的矩阵中，由SVD前k个奇异值构成的截断近似矩阵是在Frobenius范数意义下对原矩阵的最佳近似