Measure Before You Train: Benchmarks and Insights from a Personal Multilingual Inbox Routing System

The core task of a personal email agent is to make a decision about every email: what type it is, whether it's urgent, whether a reply draft is needed, whether to create a task, or whether to simply ignore it. Yet the error rates of such systems are rarely measured systematically—because the stored labels are often generated by the system itself, lacking independent ground truth.

A recent study introduced Foyer, a self-hosted personal inbox agent, along with a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, and human labeling within the agent review loop) designed to obtain reliable ground truth before any training. The study compared nine systems on 739 real Italian/English emails (231 unbiased samples, 500 adversarial samples), including production keyword rules, a TF-IDF + logistic regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B).

Study source screenshot

Key Findings: Rules Fail and the Baseline Wins

Production Rules Carry Zero Information

The most striking result: the production keyword rules achieved only 28% accuracy on the "message type" field, while the majority-class baseline is 58%. This means the field—the most prominently displayed in the interface—carries no useful information whatsoever. This finding directly challenges the common assumption that "rule-driven is good enough."

The Logistic Regression Baseline Leads Across the Board

Trained on about 200 labels, TF-IDF + logistic regression achieved the highest accuracy on every field: type 0.79, urgency 0.74, needs-reply 0.83, and is-noise 0.90. However, it never predicted the two rarest classes, and on the type field it was statistically tied with Gemma 4. Few-shot prompting did not close this gap.

This result sends a counterintuitive signal: when data is limited, classical machine learning remains a strong baseline, and the few-shot capabilities of generative LLMs do not automatically translate into routing accuracy.

TF-IDF + logistic regression is a classic and efficient combination for text classification. TF-IDF (term frequency–inverse document frequency) represents each email as a sparse vector: term frequency measures a word's importance within a single email, while inverse document frequency down-weights words that appear frequently across the entire corpus (such as "the" or "of"), thereby highlighting discriminative vocabulary. Logistic regression is a linear classifier that learns weights for each word feature and outputs class probabilities. Its advantages: with limited labels (e.g., about 200), it has few parameters and resists overfitting; training and inference are fast; and the weights are interpretable. The study shows that this simple baseline outperforms few-shot generative LLMs on multiple fields, demonstrating that in data-scarce scenarios, classical feature engineering plus linear models remain competitive.

Relationship Inference: Belongs in the Contact Graph, Not the Prompt

The study also revealed an important distinction: a sender's relationship (client, vendor, service) can be recovered from other emails from the same contact with 98.6% accuracy (provided that contact has appeared before), but no model can infer this information from a single email.

This means relationship information should be stored in a contact graph rather than stuffed into the prompt each time. For the design of personal email agents, this is an architectural guiding principle: persistent, structured context is more reliable than ad-hoc prompt engineering.

A contact graph is a structured knowledge representation with contacts as nodes and relationships (such as client, vendor, service) as edges. In a personal email agent, it persists independently of any single email and can accumulate information across emails. For example, once multiple emails from the same sender have been observed, the system can infer the relationship type and store it in the graph. Unlike stuffing relationship information into every prompt, a contact graph provides stable, queryable context, avoiding repeated inference and prompt bloat. The study's findings show that relationship information can be recovered from other emails from the same contact with 98.6% accuracy but cannot be inferred from a single email, empirically supporting the architectural choice of externalizing relationships into a graph.

Accuracy Is Not a Product Metric: The Trap of Composing Actions

The final finding is especially critical: accuracy is not a product-level metric. When classification results are composed into actions, the most accurate system ignored action-requiring emails most frequently—its ignore rate was 18%, compared with just 4% for the best LLM.

However, when a threshold was selected based on cost and applied to its noise probability, that ignore rate was halved, making it the lowest-cost system. This shows that in real products, the choice of decision threshold must align with business costs rather than pursuing classification accuracy alone.

A decision threshold is the cutoff that converts a classifier's output probability into a concrete action. For example, an agent might set: when the "noise" probability exceeds a certain threshold, ignore the email directly; otherwise, place it in the pending queue. The choice of threshold directly affects the trade-off between two types of errors (missing action-requiring emails vs. mishandling noise emails). In the study, the most accurate system's default threshold caused 18% of action-requiring emails to be ignored; after reselecting the threshold based on business costs, the ignore rate was halved and overall cost was lowest. This shows that product metrics (such as miss cost and processing cost) should guide threshold tuning rather than pursuing classification accuracy alone. Aligning thresholds with costs is a key step in turning a classification model into product actions.

Methodological Takeaway: Measure First, Then Train

Foyer's measurement protocol itself deserves attention. Stratified sampling ensured representativeness of rare classes; adversarial sampling (500 emails) exposed system performance on hard examples; frozen snapshots and thread-grouped splits prevented data leakage; and pre-registered hypotheses reduced bias from post-hoc attribution.

Together, these practices point to one principle: before training any model, establish reliable measurement infrastructure first. For highly personalized, label-scarce scenarios like personal email agents, this principle is especially important.

Conclusion

Through a rigorous comparative experiment, this study shows that on the task of routing a personal multilingual inbox, the classical baseline (logistic regression) remains strong with limited labels, and the few-shot advantage of generative LLMs is not significant; relationship information should be externalized into a contact graph; and accuracy must give way to product metrics aligned with cost. For developers building personal AI agents, the biggest takeaway may be: measure first, then train—otherwise you may be optimizing a field that carries no information at all.

One-line Summary

A personal email agent study reveals that rules fail, logistic regression leads, relationships belong in a graph, and accuracy must give way to cost metrics.

Paragraph Summary

A study of the personal email agent Foyer compared nine systems on 739 real emails. Key findings: production keyword rules achieved only 28% accuracy on message type, below the 58% majority-class baseline, carrying almost no information; with about 200 labels, TF-IDF + logistic regression had the highest accuracy on all fields (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), and few-shot generative LLMs failed to close the gap. Sender relationships can be recovered from other emails from the same contact with 98.6% accuracy but cannot be inferred from a single email, so they should be stored in a contact graph rather than in prompts. Moreover, accuracy is not a product metric: the most accurate system, once results were composed into actions, had the highest rate of ignoring action-requiring emails (18% vs. 4% for the best LLM), but cost-aligned threshold selection halved that ignore rate and made it the lowest-cost system. The study emphasizes the methodological principle of "measure first, then train."