The Linguistic Structure of Dependency-Conditioned Embeddings: New Findings from a Geometric Perspective

Research reveals that the global geometry of dependency-conditioned embeddings is dominated by POS distributions, local lexical similarity is partially preserved but neighborhoods change significantly, and PPMI is more stable than SGNS.
This study analyzes the global and local geometric properties of dependency-based lemma–relation embeddings using two PPMI and two SGNS representations trained on WikiText-103. At the global level, dependency relation centroids are highly aligned with POS distributions and remain robust after within-lemma centering, indicating that macroscopic structure is mainly driven by syntactic categories. At the local level, broader lexical similarity is partially preserved across dependency relations, but immediate neighborhoods change significantly, reflecting polysemy and context specificity. Method comparison shows that PPMI outperforms SGNS on cross-relation consistency, while cross-method consistency varies by relation type. The study emphasizes that representation choice affects the nature of the linguistic information captured, providing geometric insights for the design and interpretability of syntax-aware word representations.
Dependency-based distributional models can capture syntax-driven lexical similarity, but the geometric organization of dependency-conditioned lexical representations remains underexplored. A recent study provides an in-depth analysis of the global and local geometric properties of dependency-based lemma–relation embeddings, revealing systematic linguistic organization and sensitivity to representation choices.

Research Methods: Four Representations, Two Levels
The study uses four representations trained on WikiText-103: two count-based PPMI configurations and two negative-sampling Skip-Gram (SGNS) configurations. The analysis is conducted at both global and local levels.
At the global level, the authors analyze dependency relation centroids, control for the influence of lexical identity via within-lemma centering, and examine the alignment of centroids with part-of-speech (POS) distributions. At the local level, they evaluate cross-relation geometric consistency, cross-method consistency, lexical similarity, and polysemous usage.
PPMI (Positive Pointwise Mutual Information) is a count-based word representation method. It first counts the co-occurrence frequencies of words and contexts (here, dependency relations), then computes pointwise mutual information PMI = log( p(w,c) / (p(w)p(c)) ), and truncates negative values to 0 to obtain the PPMI matrix. Each row of this matrix can be viewed as a lemma–relation embedding. The PPMI method requires no training and directly reflects the strength of co-occurrence statistics; it is sensitive to high-frequency events, but the matrix is high-dimensional and sparse.
SGNS (Skip-Gram with Negative Sampling) is an efficient training variant of word2vec. It learns low-dimensional dense embeddings by predicting whether a center word and a context word (here, a dependency relation) co-occur, and uses negative sampling to reduce computational complexity. Unlike the explicit counts of PPMI, SGNS implicitly factorizes the co-occurrence matrix through a neural network, and the geometric properties of its embedding space are greatly affected by training hyperparameters (such as dimensionality, window size, and number of negative samples). The two may capture different aspects of linguistic information when representing dependency-conditioned words.
Global Geometry: POS Distribution Dominates Structure
The results show that, across all representations, the global geometric structure is strongly influenced by the POS distribution of contexts. This means that the macroscopic organization of dependency relation embeddings primarily reflects syntactic category information rather than specific lexical semantics. After controlling via within-lemma centering, the dependency relation centroids remain highly aligned with POS distributions, indicating that this structure is robust.
This finding is consistent with the view in distributional semantics that "syntactic information has significant structure in embedding space," but further clarifies the specific manifestation under dependency-conditioned settings.
Within-lemma centering is a geometric analysis technique that controls for the influence of lexical identity. For each lemma (e.g., "run"), its embedding vectors under different dependency relations (e.g., "run-obj", "run-subj") form a local point set. Within-lemma centering means subtracting the mean of all vectors for that lemma from each vector, thereby eliminating the global positional offset of the lemma itself and retaining only the relative differences caused by different dependency relations. In this way, when analyzing dependency relation centroids (i.e., the average of all lemma vectors under each relation), the commonality brought by lexical identity is removed, and the differences between centroids more purely reflect the structure of dependency relations (or POS distributions). This operation helps determine whether the global geometry is driven by syntactic categories rather than specific words.
Local Geometry: Lexical Similarity and Polysemous Usage
At the local level, the study finds that broader lexical similarity structure is partially preserved across dependency relations, but immediate neighborhoods change significantly, highlighting differences among usages. This indicates that although dependency relation embeddings can capture some general lexical similarity, at a fine-grained level, lexical representations under different dependency relations diverge markedly, reflecting polysemy and context specificity.
Cross-relation geometric consistency measures whether, for the same pair of lemmas, the similarity (e.g., cosine similarity) between their embedding vectors under different dependency relations remains stable. For example, if lemmas A and B are similar under the "subject" relation, are they also similar under the "object" relation? High consistency means that dependency relation embeddings preserve a general semantic similarity structure among lemmas; low consistency indicates that local neighborhoods are significantly reorganized under different relations, possibly reflecting polysemy or relation specificity. This metric is quantified by comparing similarity matrices across multiple relations and is a key local geometric measure for assessing whether representations capture stable lexical structure.
Method Comparison: Differences Between PPMI and SGNS
When comparing different representation methods, PPMI-based representations outperform SGNS-based representations on cross-relation consistency, while cross-method consistency varies by relation type. This finding suggests that count-based PPMI methods may more stably capture structural consistency across dependency relations, whereas SGNS methods may be more sensitive to local changes.
The study also points out that cross-method consistency differs across dependency relations, indicating that the choice of representation method affects the nature of the linguistic information captured.
Implications and Outlook
These findings reveal systematic linguistic organization of dependency-conditioned embeddings across multiple geometric scales and emphasize the impact of representation choice on analysis results. For researchers and practitioners working on syntax-aware word representations, this work provides an in-depth understanding of the geometric properties of embedding spaces, helps design more effective representation methods, and promotes discussion of the interpretability of distributional semantic models.
Future work can further explore other languages and more complex dependency relations, as well as how to incorporate these geometric insights into downstream tasks.
One-line Summary
The study reveals that the global geometry of dependency-conditioned embeddings is dominated by POS distributions, local lexical similarity is partially preserved but neighborhoods change significantly, and PPMI is more stable than SGNS.
Paragraph Summary
This study analyzes the global and local geometric properties of dependency-based lemma–relation embeddings, using two PPMI and two SGNS representations trained on WikiText-103. At the global level, dependency relation centroids are highly aligned with POS distributions and remain robust after within-lemma centering, indicating that macroscopic structure is mainly driven by syntactic categories. At the local level, broader lexical similarity is partially preserved across dependency relations, but immediate neighborhoods change significantly, reflecting polysemy and context specificity. Method comparison shows that PPMI outperforms SGNS on cross-relation consistency, while cross-method consistency varies by relation type. The study emphasizes that representation choice affects the nature of the linguistic information captured, providing geometric insights for the design and interpretability of syntax-aware word representations.
Related articles

New Breakthrough in Model Merging: Interference Bounds and Curvature-Guided Training
This article interprets new theoretical advances in model merging: characterizing merging excess loss via an interference functional, proposing curvature-guided regularized training, and analyzing its experimental performance and limitations on Transformers.

A New Breakthrough in Differential Privacy: How to Achieve Instance-Optimal Moment Estimation?
This article interprets research on instance-optimal moment estimation under the add-remove model of differential privacy, covering instance-dependent error metrics, tight lower bound proofs, and new mechanism design, comparing with existing methods to show utility gains and discussing implications for privacy-preserving practice.

FamilySpace: From Emergent Chemical Semantics to Controllable Crystal Generation
Crystal generation latent spaces spontaneously cluster by elemental family? FamilySpace leverages this emergent chemical semantics to enable composition-controllable crystal generation via multi-source flow models, balancing quality and stable exploration on MP-20 and MPTS-52.