HAIDL: Upgrading Human-AI Interaction into Reproducible Structured Data

AI assistants are now deeply embedded in computational notebooks, scientific workflows, software engineering tools, and learning systems, yet most tools still treat human-AI interaction as a one-off "prompt-response" exchange that is discarded after use. This approach makes reproducibility, interpretability, behavioral analysis, and adaptive support all difficult. A new paper proposes that human-AI interaction should be treated as a new category of data requiring formal representation, and introduces an interaction description language called HAIDL along with a companion JupyterLab middleware named HAIDL-X.

Why Human-AI Interaction Needs "First-Class Data" Status

Currently, interaction records with AI assistants in environments like JupyterLab typically exist only in chat history, lacking structured context. Which notebook cell the user was in, what code they ran, what error they encountered, what advice the AI gave, and whether the user adopted it—this information is either scattered or not recorded at all.

The paper points out that this "transient exchange" model directly undermines several key capabilities: reproducibility (inability to replay the complete AI-assisted process), interpretability (inability to trace the contextual basis of AI suggestions), behavioral analysis (inability to quantify how users seek help), and adaptive support (systems cannot adjust strategies based on historical interactions). Elevating interactions to persistent data products is the only way to systematically address these issues.

HAIDL: Describing Each Interaction with BNF-Like Syntax

The core idea of HAIDL (Human-AI Interaction Description Language) is to encode each interaction as a structured object. This object contains multiple fields: computational context, user request, prompt payload, AI response, feedback, problem context, provenance information, and model metadata.

Drawing on the ideas of Backus-Naur Form (BNF), the paper designs a concrete grammar for HAIDL. This means interaction records are no longer free text but parseable, queryable structured data. Each field has clear semantic boundaries, allowing interactions produced by different tools at different times to be compared and aggregated under the same framework.

The direct benefit of this formalization is that interaction objects can be stored, retrieved, replayed, and used as input for downstream analysis. The authors emphasize that this is the foundation for treating human-AI collaboration as "structured, queryable, replayable, and analyzable" data.

Backus-Naur Form (BNF) is a meta-syntax notation used in computer science to describe the grammar of programming languages, proposed by John Backus and Peter Naur in the late 1950s, originally for defining the grammar of ALGOL 60. BNF decomposes grammatical structures into terminal symbols and non-terminal symbols through a set of production rules, typically in the form <non-terminal> ::= <expression>. Its core value lies in its ability to precisely and unambiguously characterize the legal structure of a language with minimal notational conventions, making grammars machine-parseable and facilitating sharing of a single specification across different implementations. Borrowing BNF ideas in HAIDL means that the field structure, nesting relationships, and value constraints of interaction objects can be explicitly written with concise rules, transforming interaction records from free text into verifiable, parseable structured data and providing a common grammatical basis for cross-tool interoperability.

HAIDL-X: Runtime Implementation in JupyterLab

To validate this model, the authors built HAIDL-X—a JupyterLab middleware. It is responsible for instantiating HAIDL in real interactions: capturing user and bot states, constructing context-aware prompts, interfacing with AI providers, collecting feedback, and storing interactions in a repository.

The value of HAIDL-X lies in bringing the formal model into actual workflows. Users do not need to manually fill in structured fields; the middleware handles capture and storage in the background. For researchers, this means obtaining high-quality interaction data without disrupting users' natural operations.

JupyterLab is the next-generation web-based interactive development environment of the Jupyter project, extending the classic Notebook into a workbench with multi-panel layouts: the left side can simultaneously open file browsers, consoles, and terminals, while the central area can place notebooks, text editors, data viewers, and other components side by side. Its architecture hinges on the "extension system"—JupyterLab itself is a container separating the kernel from the frontend, and third parties can mount TypeScript/JavaScript extensions to the frontend to listen for events such as cell execution, kernel state, and document switching, and inject custom panels or buttons into the UI. This "subscribable events, pluggable UI" mechanism is precisely the technical prerequisite that enables HAIDL-X to seamlessly capture user-AI interactions in the background as middleware, bind cell context, and write structured records to the repository.

The Learning Layer: From Interaction Data to Reproducible Behavioral Metrics

The paper adds a learning layer on top of HAIDL that maps interaction objects and notebook execution traces to reproducible user behavioral metrics. These metrics include engagement, performance, help-seeking behavior, error-oriented usage patterns, and learning personas (user profiles).

The significance of this layer is that it transforms raw interaction records into interpretable indicators. For example, by analyzing the relationship between user requests and subsequent code execution, one can determine whether users solve problems independently or rely on AI; through feedback fields and error contexts, one can identify which interactions genuinely helped the user. Because the underlying data is structured, these metrics are themselves reproducible—another researcher using the same interaction repository should obtain consistent results.

Notebook execution trace refers to the temporally ordered sequence of cell execution events recorded in a notebook, including the input code of each cell, execution order number, execution timestamp, kernel state changes, and output or exception information. Unlike static notebook files, execution traces preserve temporal information about "what actually happened," such as the same cell being repeatedly modified and re-run, reverting to previous cells after errors, or skipping certain cells. In the HAIDL framework, execution traces are aligned with interaction objects, enabling causal chains such as "at which step the user encountered an error, when the AI suggestion appeared, and whether the user modified and re-ran after the suggestion" to be reconstructed—this is the underlying evidence on which the learning layer depends to compute metrics like help-seeking behavior, error coping strategies, and engagement.

Implications for the Design of AI Collaboration Tools

This work sends a clear signal: the product design of AI assistants should not only focus on the quality of individual responses but also on the data architecture of interaction records. When interactions become first-class data, tools can support replay debugging, behavioral auditing, personalized adaptation, and even cross-session learning transfer.

For teams building AI coding assistants or educational AI products, HAIDL's approach is worth adopting: defining interaction data schemas upfront is far more reliable than piecing together context from logs after the fact. Of course, this solution currently focuses on the JupyterLab ecosystem, and whether it can be extended to broader IDEs and conversational interfaces remains to be further validated.

One-line Summary

The paper proposes treating human-AI interaction as first-class data, using the HAIDL language and JupyterLab middleware to achieve structured recording and analysis.

Paragraph Summary

This article introduces a study that elevates human-AI interaction to "first-class data." Currently, interactions with AI assistants in environments like JupyterLab are mostly transient exchanges, with context scattered across chat history, undermining reproducibility, interpretability, behavioral analysis, and adaptive support. The paper proposes HAIDL, an interaction description language that draws on BNF ideas to encode each interaction as a structured object containing computational context, user request, prompt payload, AI response, feedback, problem context, provenance, and model metadata, making it storable, retrievable, replayable, and analyzable. The companion HAIDL-X, as JupyterLab middleware, automatically captures and stores interactions in the background, reducing collection costs. Its learning layer maps interaction objects and notebook execution traces to reproducible metrics such as engagement, performance, help-seeking behavior, error coping strategies, and learning persona. This solution provides design reference at the data architecture level for AI collaboration tools, but its current scope is limited to the JupyterLab ecosystem.