Sultry Is Not a Vector: The Linear Probe Mirage

TL;DR: A linear probe that predicts “sultry” from embeddings doesn’t prove a semantic subspace exists; it may only prove that high-dimensional geometry and dataset artifacts made a shortcut linearly extractable.

The vector is lying.

Take sultry. It’s humid weather, smoky vocals, noir lighting, erotic charge, tropical air, gendered description, genre convention, and context-dependent social coding. The linear representation hypothesis turns that mess into a clean claim: somewhere in embedding space, “sultriness” lives as a direction, or at least as a low-dimensional linear subspace. Train a probe, get high accuracy, draw the arrow. Done. Except a linear read-out only proves extractability under a sampling process. It doesn’t prove the model has learned a stable concept, uses that concept causally, or would preserve it under domain shift. In high dimensions, separability is cheap.

This is the part AI hype keeps flattening. Benchmark performance becomes ontology. A demo becomes a capability claim. A classifier boundary becomes “understanding.” We saw the same pattern at product scale when McDonald’s ended its IBM drive-through AI trial after viral ordering failures: polished automation can look competent until the operating distribution changes. Probing classifiers have a quieter version of that problem. If “sultry” examples come disproportionately from romance blurbs, weather reports, nightclub reviews, or caption datasets with predictable annotation habits, the probe can learn source domain, token frequency, sentence length, punctuation, gendered adjectives, or co-occurrence with words like “humid,” “red,” “jazz,” and “night.” Not sultriness. The shadow of sultriness.

The spectral diagnostics usually tell on the story. Let embeddings be (X \in \mathbb{R}^{n \times d}) and a linear probe use (W \in \mathbb{R}^{c \times d}). If (W) has low stable rank, (|W|_F^2 / |W|_2^2), and its top singular vectors align with dominant principal components of (X), the probe may be reading corpus anisotropy: topic, format, frequency, dataset origin, or prompt template residue. If the spectrum is diffuse, that isn’t automatically better; it may mean the probe is pooling many weak artifacts into one deceptively robust boundary. Either way, accuracy is not causal evidence. It’s separability.

A serious audit should treat the probe as a suspect instrument, not a certificate of meaning:

• Rank and spectrum: Inspect singular values of probe weights and representation covariance; “clean” low-rank directions can be nuisance axes, while high effective rank can hide aggregated shortcut features.
• Counterfactual invariance: Hold topic fixed while changing connotation, then hold connotation fixed while changing domain. A real “sultry” representation should survive both interventions.
• Margin and neighborhood stability: If cosine margins, nearest neighbors, or classifier decisions flip under paraphrase, debiasing, principal-component removal, or adversarial splits, the alleged concept was geometry on a knife-edge.

Cosine similarity doesn’t rescue the claim. For random unit vectors in (d) dimensions, dot products concentrate around zero with variance about (1/d), and Euclidean distances concentrate near (\sqrt{2}). Not literally identical. But close enough that local neighborhoods become unstable and tiny anisotropies can dominate retrieval. Real embedding spaces are not isotropic spheres; they’re hub-heavy, frequency-shaped, and full of global components learned from web-scale regularities. That makes cosine similarity socially and statistically fragile. “Sultry” might sit near “humid” in one slice, “seductive” in another, “lounge” after a preprocessing change, and “tropical” after a corpus shift. The neighborhood looks semantic because we narrate it that way.

The Johnson–Lindenstrauss lemma is often misused here, so be precise. It says that for (n) fixed points, a random projection into (m = O(\log n / \varepsilon^2)) dimensions preserves pairwise Euclidean distances within a ((1 \pm \varepsilon)) factor. That’s a distance guarantee, not a meaning guarantee. If two semantic relations differ by less than the allowed distortion — for example, if (|h{\text{sultry}} - h{\text{humid}}|) and (|h{\text{sultry}} - h{\text{seductive}}|) are separated by only a small margin — projection can preserve almost all pairwise distances while still flipping nearest-neighbor order, analogy structure, or a linear probe’s boundary. JL doesn’t prove dimensionality reduction always destroys semantics. It proves something more uncomfortable: if your semantic claim depends on fragile, low-margin geometry, then preserving distances is not the same as preserving meaning.

The responsible conclusion is not that embeddings contain no useful semantics. They do. The conclusion is narrower and more important: linear probe accuracy is too weak to justify claims of conceptual understanding, especially in systems that affect work, education, care, benefits, safety, or public trust. If we want AI that strengthens a free and secure society instead of just producing better funding slides, we need causal tests, counterfactual datasets, spectral audits, and failure reporting before we turn separability into authority. Stop asking whether a probe can find “sultry”; start asking what shortcut it used to get there.