JSONL

The JSONL (JSON Lines) format treats each line as an independent JSON object, supports streaming append writes without needing to re-parse the entire file, and write operations are O(1).

In JSONL format, each line is an independent and complete JSON object, supporting streaming reads, parallel processing, and incremental appending; Hugging Face's Datasets library natively supports loading it.

Pi Durable currently supports three storage backends: SQLite, JSONL file storage, and in-memory storage, and with in-memory storage the data is lost once the process is closed.

In Pi Durable, SQLite suits scenarios that require querying state by condition and guaranteeing transactional atomicity, while JSONL suits log-style appending, post-hoc auditing and quick manual inspection.

Pi Durable writes to persistent storage on every tool call, which brings additional I/O overhead and storage growth; for Agents with high-frequency calls, JSONL files will rapidly expand, and SQLite will also face write pressure.

Pi Durable writes the state checkpoint of each task and each tool call to SQLite or JSONL persistent storage.

The storage format chosen is append-only JSONL, which withstands hard shutdowns better than repeatedly rewriting a JSON file and is also easier to grep and diff.

A run ledger is an append-only, chronological log file that writes one line per model call, always recording model identifier, endpoint, timestamp, token usage, and latency to support after-the-fact auditing.

The ledger script ledger.py depends only on the Python standard library (Python 3.11+), appending one line of JSON per attempt, so retries are naturally billed.

The example writes accepted events to a JSONL audit log but deliberately does not store the sender and recipient phone numbers, recording only the event type, message ID, source check result, and whether charges may be incurred.

The JSONL log scenario implies a contract: each line is self-contained, fields may be absent, and the parser must remain tolerant of unknown types or newly added fields, and must not fail the entire batch because of an unknown record.

The harness writes one line of JSONL per drafting attempt, containing fields such as task_id, placement, attempt_no, prompt_tokens, completion_tokens, wall_ms, ok, error_class, and sql_sha256.

Using JSONL append writes instead of JSON overwrite writes is the standard practice for idempotent tasks: each line of JSONL is independent, and an interrupted write only loses the last line; with overwrite-style JSON, an interrupted write corrupts the entire file and progress returns to zero.

JSONL append writing carries almost no performance overhead, requires no complex database transactions, and does not block the agent's main workflow.

JSONL records state change logs in append-only fashion, suitable for auditing and sequential replay.

The system selects different persistence technologies based on access patterns, adopting a hybrid storage approach combining ChromaDB, JSONL, and SQLite.

The practice of recording every state transition yields a complete audit trail, making it easy to trace back the state changes of any agent at any point in time.

The core persistence pattern is: write one JSONL audit record per state transition, and read the tail of the log on each restart.

Agent Standup directly reads JSONL-format session files written by Claude Code to the ~/.claude/projects directory and tracks the current session in real time.

The advantage of JSONL over standard JSON arrays is efficient append writing and line-by-line processing; the disadvantage is poor human readability.