PCA

PCA dimensionality reduction finds the projection axes of maximum data variance by performing eigendecomposition on the data covariance matrix, thereby compressing dimensions while retaining the most information.

SVD is the mathematical core of PCA dimensionality reduction, recommender system matrix factorization, and the LoRA low-rank adaptation fine-tuning method.

PCA (Principal Component Analysis) is a classic unsupervised dimensionality reduction technique that finds the directions of greatest variance in data (principal components) by computing the eigenvalues and eigenvectors of the data's covariance matrix.

PCA was first proposed by Karl Pearson in 1901 and later developed and refined by Harold Hotelling in 1933.

PCA is equivalent to performing eigenvalue decomposition on the data covariance matrix; if SVD is applied to the centered data matrix X to obtain X=UΣVᵀ, then the right singular vectors V are the principal component directions of PCA, and the squared singular values divided by the number of samples are the variances corresponding to each principal component.

Principal Component Analysis (PCA) involves both eigenvalue decomposition from linear algebra and variance maximization from statistics.

The abliteration technique constructs a contrastive dataset of harmful requests versus normal requests, uses PCA or mean-difference methods to locate the principal direction vector of refusal behavior, then orthogonally removes it from the weight matrix via projection—requiring no gradient updates or retraining, and can be completed in a few minutes on a consumer-grade GPU.

After zero-training PCA injection, audio-to-audio tasks improve: gender clustering jumps from 0.05% to 63.1%, artist retrieval rises from 55.2% to 93.8%, emotion recognition from 19.7% to 40.2%, and environmental sounds from 38.8% to 56.3%.

Using PCA to compress layers 2 and 6 to 256 dimensions (fitted on 1400 clips from 4 external audio datasets, without using MAEB/MSEB data), injecting into the null space and normalizing, text embeddings remain bit-for-bit unchanged, and text-to-audio retrieval changes by within 0.05 points.

PRSR's low-rank subspace estimation applies SVD or PCA to the matrix formed by the residual vectors of parallel sentence pairs, takes the top k singular vectors to span the subspace, and projects the final-layer token hidden states onto the orthogonal complement of that subspace.

Common dimensionality reduction techniques such as t-SNE, UMAP, and PCA can project high-dimensional embedding vectors onto two-dimensional or three-dimensional planes to observe clustering structures.

Traditional methods such as PCA and TRPCA tend to flatten the data or ignore its inherent multimodal tensor structure, thereby failing when confronted with complex faults involving sensor-time correlations

In the same case study, TRPCA and PCA-based anomaly detectors achieved only 0.69 or lower macro F1 scores

The author tried three dimensionality reduction methods—PCA, UMAP, and tSNE—and ultimately concluded that tSNE produced the best structure.

PCA is a linear dimensionality reduction method that retains the directions of greatest global variance; it is computationally fast but has limited expressiveness for nonlinear structures.

The project includes PCA and t-SNE dimensionality reduction analysis implemented in NumPy, allowing layer-by-layer viewing of the test set's distribution in feature space.

PCA reduces the dimensionality of hidden activations from multiple samples of the same prompt; the first principal component captures the direction of greatest variance among the sampled results, corresponding to the style dimension along which the model varies most dramatically.

The method proceeds in three steps: repeatedly sampling a single prompt at high temperature to generate multiple completions, applying PCA dimensionality reduction to the hidden activations of the completions, and automatically assigning semantic labels to the positive and negative poles of each principal axis—all without updating model parameters.

The paper proposes a training-free, prompt-conditioned method for LLM style axis discovery that extracts style structure from model decoding variance by repeatedly sampling at high temperature on the same prompt.

The low recall indicates that relying solely on the top two principal axes with the greatest variance cannot cover all style dimensions that humans care about; extending to more principal components can improve recall but introduces noisy axes that cause precision to decline.