How an LLM Agent Audited a Slovak Corpus: Discovering Machine-Generated Content Farms

Title

LLM Agent 如何审计斯洛伐克语语料库：发现机器内容农场

Meta Description

研究利用 LLM 代理审计 FineWeb2 斯洛伐克语语料库，发现 5.32% 为机器生成内容农场，编译测试套件扫描全库仅需 0.7 秒，成本远低于传统分类方法。

Article Content

Multilingual corpus construction often relies on a one-size-fits-all pipeline, which makes language-specific failure modes hard to anticipate. A recent study tried letting an LLM act as a "corpus debugger" rather than a classifier, exploring data through a tool-calling agent and turning findings into executable tests. On the Slovak portion of FineWeb2 (26.5 million documents), three runs (each costing $2.21–2.84) uncovered seven issues across four categories of phenomena, the largest being a machine-generated content farm comprising 1,408,437 documents (5.32%).

Key Finding: Machine Content Farms Bypass Filters

The biggest issue the agent found was a machine-generated content farm accounting for 5.32% of the Slovak corpus (1,408,437 documents). More surprisingly, 86.6% of that content even passed stricter threshold filtering. This means existing quality filters have a systematic blind spot when faced with organized machine-generated content.

Blind annotation supported 6 of the 7 findings, with defect rates between 0.53 and 1.00, compared to just 0.08 for the random control group. One finding was not supported, and the research team reported this result honestly.

Content Farm refers to an organization or system that produces web content in large volumes at low cost, typically for the purpose of monetizing SEO traffic. In the LLM era, content farms have evolved to use large models to generate batches of text that looks fluent but is actually information-poor or repetitive, then fill websites through automated publishing. Such content often has a certain level of language fluency, so it can bypass filtering thresholds based on perplexity, repetition rate, or simple quality classifiers. For corpus builders, the harm of content farms lies in diluting high-quality data, introducing factual noise, and potentially causing downstream models to learn templated expressions. Identifying such content usually requires combining signals such as domain source, publishing time patterns, text diversity, and cross-document template similarity, rather than relying solely on single-document quality scores.

Method: Agent as Debugger

The research team used the LLM as a one-off debugger rather than a classifier. The agent explored the corpus through a read-only, budget-limited API and compiled each finding into an executable test, which was then validated by native speakers. The compiled test suite scans the entire corpus in just 0.7 seconds of wall time without a GPU, whereas each LLM classification pass costs between $12,000 and $90,000.

Interestingly, the best static sampling baseline validated more selectors than any agent run and at less than half the cost. The agent's core contributions were: precise prevalence estimation, precision estimation checked against sampled documents, explanations of why the pipeline missed each slice, and reusable tests.

Tool-calling Agent refers to an LLM system that can autonomously decide which external tools (such as APIs, database queries, code execution environments) to call to complete a task. Unlike using an LLM as a single-step classifier, an agent explores data through multiple rounds of an "observe—decide—act" loop and can dynamically adjust its query strategy. In this study, the agent was given a read-only, budget-limited corpus API, enabling it to sample documents, compute distributions, test hypotheses, and turn findings into executable tests. The advantage of this paradigm lies in interpretability and reusability: the agent not only provides labels but also outputs reasoning about "why the pipeline misses this slice" and test code that can run independently. Its limitations are cost and randomness, and it may not outperform a carefully designed static sampling baseline on pure classification metrics.

Blind Annotation is an evaluation method in which annotators make independent judgments without knowing the sample source or expected conclusion, in order to reduce confirmation bias. In this study, native speakers blindly annotated the seven issues found by the agent, with randomly sampled documents as a control group. The defect rate (i.e., the proportion of samples judged problematic by annotators) ranged from 0.53 to 1.00, while the control group was only 0.08, indicating that the problem slices found by the agent were indeed enriched with defects. The key to the blind design is that annotators cannot distinguish "agent-selected samples" from "random samples," thereby ensuring the objectivity of the evaluation. One finding was not supported, and the research team reported it honestly, reflecting the framework's tolerance for false positives and transparency.

Multi-source Validation and Implications

According to the research paper and accompanying agent trajectory data, the framework, Slovak test suite, validation data, and agent trajectories have all been released. This provides a low-cost, high-efficiency new paradigm for auditing multilingual corpora.

For teams building multilingual data pipelines, this study suggests that failure modes of generic recipes on specific languages require targeted auditing, and that LLM agents can serve as an effective debugging tool but should not completely replace static baseline methods. The agent's advantage lies in interpretability and the generation of reusable tests, not in pure classification accuracy.

Conclusion

This study demonstrates the potential of LLM agents in corpus auditing: discovering large-scale machine-generated content farms at extremely low cost and generating reusable test suites. At the same time, the study honestly presents the agent's limitations—static baselines perform better on some metrics. For practitioners concerned with multilingual data quality and AI-generated content detection, this work provides practical methods and tools.

One-line Summary

LLM agent found 5.32% machine-generated content farms in the Slovak FineWeb2 and generated reusable tests.

Paragraph Summary

A study used an LLM as a corpus "debugger" rather than a classifier, exploring the Slovak portion of FineWeb2 (26.5 million documents) through a tool-calling agent. Three runs (each costing $2.21–2.84) uncovered seven issues across four categories of phenomena, the largest being a machine-generated content farm accounting for 5.32% (1,408,437 documents), with 86.6% of that content passing stricter filtering thresholds. Blind annotation supported six of the seven findings, with defect rates of 0.53–1.00 versus just 0.08 for the control group. The agent compiled findings into executable tests that scan the entire corpus in only 0.7 seconds, far below the cost of an LLM classification pass. However, a static sampling baseline validated more selectors at lower cost; the agent's core value lies in interpretability and the generation of reusable tests.