Skip to main content
Version: 1.0.11

Datasets and data flow

When an agent produces a large result — thousands of rows from a query, a computed table, a time series for a chart — it does not paste those rows into the conversation. Instead the coder registers a dataset: the full data stays as a file in your S3 conversation filesystem, and the chat renders a compact preview, chart, or grid from a reference to it. This page explains the dataset-registration model, how chat reads from a dataset without loading all of it, the conversation-scoped dataset store, how datasets are rediscovered across turns, and how data moves between workflow nodes that do not share a filesystem.

Why datasets exist​

A model's context window is finite and dumping a large table into it is both wasteful and unreliable. The dataset model solves this: the coder writes the data to a file in the conversation filesystem, then registers that file as a dataset with register_artifact. The registration records lightweight metadata — a meaningful name and description, the data's shape (row and column counts, column names), a small preview, and the file's location in S3 — while the full data never enters the conversation.

The chat then renders from that reference. A table becomes a scrollable grid, a series becomes a chart, and the UI pulls only the rows or columns it needs to display, on demand, directly from S3.

How registration works​

After generating a data file in the conversation filesystem, the coder registers it as a dataset with register_artifact. The registration captures:

  • Name and description — a meaningful, human-readable label and a short description of what the dataset is, so the agent can recognise it again on a later turn.
  • Type and format — for example a table or a series, stored as CSV, JSON, or similar.
  • Shape — row count, column count, and column names.
  • A preview — a small number of rows for an at-a-glance view.
  • Location — the dataset's key in the S3 workspaces bucket, which is what every later reference resolves against.

The full dataset stays in S3 under the conversation's datasets/ prefix. What flows into chat is the reference plus the preview — not the rows.

How chat renders without loading everything​

Because the reference points at the file in S3 and the registration knows the data's shape, the chat can fetch exactly what a view needs:

  • Grids request a page of rows at a time (offset and limit), so a million-row table scrolls without ever being fully loaded.
  • Charts request only the specific columns being plotted.

The agent's textual answer stays concise — it talks about the data and points at the dataset — while the heavy data is served separately and only as needed. This is what lets an agent "return ten thousand rows" in a conversation that still reads cleanly.

The session-datasets store​

Dataset records live in a dedicated DynamoDB table in your account (the session-datasets store). Each record is keyed by the session that produced it and, for chat, also tagged with the conversation it belongs to (queryable through a conversation index). Because a conversation spans many short-lived sessions (a new one is created each time you send a message), the conversation tag is what lets the agent find datasets it registered in an earlier turn after you leave and return: it calls list_artifacts to enumerate the conversation's datasets by name and description, then load_artifact to pull the one it needs back into the working directory. These records persist for the life of the conversation/run — there is no fixed expiry, so a long, multi-step run keeps every dataset it registers available from start to finish.

Two things are worth separating here:

  • The dataset record is a lightweight pointer (metadata and reference) in DynamoDB.
  • The underlying file in the S3 conversation filesystem holds the actual data and follows the filesystem's own lifetime (see How agents run code).

Documents you upload are handled the same way on the storage side: they land under the conversation's docs/ prefix in S3, and the coder reads them with its read_document tool when an analysis needs them.

Keeping both for the duration of the run is what lets genuinely long-running analyses (deep traversals and multi-node workflows that can run for hours) reference data they produced much earlier without it disappearing mid-run.

Moving data between workflow nodes​

In a workflow, each node runs in its own sandbox session and its own filesystem prefix — nodes do not share a working directory. So when one node produces data that the next node needs, the data has to be moved explicitly.

The mechanism is load_artifact. When an upstream node registers a dataset, its output includes the dataset's identifier and S3 location. A downstream node calls load_artifact with that reference, which downloads the file from S3 into the downstream node's live working directory. The next piece of code in that node can then open the file directly — for example reading it into a dataframe — as if it had produced the file itself.

This keeps node workspaces isolated by default — which is what preserves per-node and per-user separation — while giving you a clear, explicit way to hand data forward through a pipeline. Because the transfer is a copy from your own S3 into another in-account session, the data never leaves your account.

Where to go next​