[KongchangAI]
· 2 min read· 1,000 words

GHAgentCI: The First Large-Scale Dataset of CI Coding Agent Invocations

GHAgentCI: The First Large-Scale Dataset of CI Coding Agent Invocations

GHAgentCI releases the first large-scale dataset of CI coding agent invocations, with 15K+ workflow histories and an open-source detection tool.

GHAgentCI is the first large-scale dataset on coding agent integration in GitHub Actions, screening 196K+ repositories down to 6K+ with agent traces and building 15K+ workflow histories and 205K+ evidence records. The open-source cai-miner tool achieves 96.1% and 98.8% accuracy for dedicated-action and GitHub Agentic Workflow matches. Preliminary analysis reveals a shift toward command-line and Agentic Workflow integration, varied security safeguards, and frequent workflow removal.

Title

GHAgentCI: The First Large-Scale Dataset of CI Coding Agent Invocations

Meta Description

GHAgentCI is the first large-scale dataset of CI coding agent invocations, covering 15K+ workflow histories and 205K+ evidence records, with the open-source tool cai-miner enabling high-precision detection and providing a key resource for studying coding agent integration.

Article Content

Coding agents are rapidly being integrated into GitHub Actions, taking on tasks such as PR review, issue triage, and code modification. However, these agents' integration patterns are often buried in routine CI files and are difficult to identify with conventional methods. The recently released GHAgentCI dataset fills this gap, providing the first large-scale resource for studying the long-term integration of coding agents in CI.

GHAgentCI dataset release

Dataset Scale and Composition

GHAgentCI screened 196K+ GitHub repositories down to 6K+ repositories containing traces of coding agents, and constructed 15K+ workflow histories covering 159K+ revisions and 205K+ agent evidence records. Each piece of evidence is linked to a workflow trigger and declared execution constraints, enabling researchers to trace the behavioral context of agents in CI.

The core value of this dataset lies in its structured design: it not only records the presence of agents but also preserves their association with the workflow lifecycle. This provides a foundation for analyzing the patterns, evolution, and impact of agent integration.

Workflow triggers in GitHub Actions specify the events under which a workflow runs automatically, such as pull_request, issue_comment, or schedule. For coding agents, triggers often determine when they are invoked: an agent triggered by PR events may be responsible for reviewing or modifying code, while an agent triggered by comments may respond to user instructions.

Declared execution constraints refer to the restrictions imposed on an agent's runtime environment through fields such as permissions, env, timeout-minutes, or conditional statements in the workflow. These constraints reflect the degree of control integrators exercise over an agent's permissions, runtime duration, and accessible resources, and are also an important basis for assessing whether security safeguards are adequate.

Detection Method and Validation

The research team open-sourced the extraction tool cai-miner, which identifies agent-related traces through an extensible evidence catalog, including action references, shell commands, and workflow conventions. To validate accuracy, the team manually inspected 576 stratified records: dedicated-action matches achieved 96.1% accuracy, and GitHub Agentic Workflow matches achieved 98.8% accuracy. Other evidence types had lower accuracy, but the overall method still demonstrated high reliability.

The extensibility of cai-miner means that as new agent tools emerge, the evidence catalog can be continuously updated to keep detection capabilities at the forefront.

GitHub Agentic Workflow is a mechanism introduced by GitHub that lets AI agents participate in repository automation in a declarative manner. Developers can define an agent's trigger conditions, available tools, and permission scope in workflow files, and the agent autonomously completes tasks such as issue triage, code modification, or PR review during CI runtime. It differs from ordinary action references in that it usually carries dedicated syntax markers and runtime configuration, making it a key type of evidence for cai-miner to identify agent traces.

Dedicated-action refers to GitHub Actions designed and published specifically for coding agents, such as third-party actions for automatically fixing code or generating review comments. Such actions usually carry obvious agent semantics in their names, descriptions, or input parameters, allowing detection tools to match action references precisely. Because they follow relatively fixed publishing patterns, match accuracy is high, making them a high-confidence source of evidence in the dataset.

Exploratory analysis of the dataset shows that the integration of coding agents is shifting: command-line and GitHub Agentic Workflow integrations are gradually increasing, declared security safeguards vary, and detected agent workflows are frequently removed. These findings suggest instability in practice and ongoing adjustments to agent behavior.

GHAgentCI makes longitudinal research on the integration of coding agents into CI possible, and future work can build on it to analyze the impact of agents on development processes, security risks, and the formation of best practices.

Longitudinal research refers to repeated observation of the same set of subjects over an extended period to reveal their evolutionary trajectories and causal relationships. GHAgentCI preserves historical revisions and evidence records of workflows, enabling researchers to track the full lifecycle of agent integration from introduction and adjustment to removal, rather than merely a cross-sectional snapshot at a single point in time. This design supports analysis of changes in integration approaches over time, the survival rate of agent workflows, and the evolution of security safeguards, providing a methodological foundation for understanding the long-term impact of coding agents in CI.

Significance and Outlook

As the first large-scale dataset of coding agent invocations in CI, GHAgentCI not only provides a data resource but also contributes a reusable detection tool. It is expected to advance research at the intersection of software engineering and AI agents, helping developers and organizations use coding agents more safely and efficiently.

The open-sourcing of the dataset and tool will promote community collaboration and accelerate deeper understanding of agent integration patterns. As coding agents become more widespread, such research is essential for ensuring the reliability and security of CI pipelines.

One-line Summary

GHAgentCI releases the first large-scale dataset of CI coding agent invocations, featuring 15K+ workflow histories and an open-source detection tool.

Paragraph Summary

GHAgentCI is the first large-scale dataset on coding agent integration in GitHub Actions, screening 196K+ repositories down to 6K+ repositories containing agent traces and constructing 15K+ workflow histories, 159K+ revisions, and 205K+ evidence records. The research team open-sourced the extraction tool cai-miner, which identifies traces such as action references, shell commands, and workflow conventions through an extensible evidence catalog; manual validation shows that dedicated-action and GitHub Agentic Workflow matching achieve 96.1% and 98.8% accuracy, respectively. Preliminary analysis finds that integration approaches are shifting toward command-line and Agentic Workflow, security safeguards vary significantly, and agent workflows are often removed. The dataset provides a foundational resource for longitudinal research on the patterns, impacts, and risks of coding agent integration into CI.

#GHAgentCI#coding agents#GitHub Actions#CI dataset#cai-miner
Share:

Related articles