Datasets

Two views of the same data

A dataset holds two things at once, and the difference is the whole point.

The current view is a table of logical records: one row per product, listing, or article, showing its latest values. This is what you look at, filter, sort, and export. It is what a spreadsheet would give you.

The observation history is append-only: every time a record was seen, and exactly what it looked like on that occasion. Nothing is overwritten. A run does not replace yesterday's data; it appends today's observation and updates which one is current.

That is why Ray9 can answer "what did this cost last Tuesday?" and "when did this go out of stock?" — questions that a spreadsheet, having overwritten the cell, cannot answer at all.

Record keys

A record needs a stable identity, or every run produces a fresh pile of rows and nothing can be compared with anything.

The record key is what provides it. You derive it from:

• one or more fields — a SKU, a product ID, a listing reference,
• the canonical URL of the item,
• or an explicit expression you define.

Get this right and everything downstream works: upserts land on the correct record, history accumulates in the right place, and monitoring can tell you a price changed rather than that an old record vanished and a new one appeared. Get it wrong and you have duplicates.

Ray9 also deduplicates identical observations using a normalized content hash. Re-running a Flow against an unchanged page does not manufacture a fake change event or a redundant version — but it still records that the record was observed in that run.

Schema versions

A dataset's schema is versioned, and every observation records the schema version that validated it.

This makes evolution safe. When you add an optional field, that is a compatible change and existing records are untouched. When you change a field's type, make one required, or change the record key, that is a change that requires migration — and Ray9 says so, with a compatibility preview, before you commit to it.

Because each observation is tied to the schema that validated it, an old record is never retroactively "invalid" because you changed the rules afterwards. It was valid under the schema in force at the time, and it says so.

Provenance on every value

Every observation carries where it came from:

Attach the run and you can reach the attempt behind it, and through that the evidence: the captured HTML, the screenshot, the trace, the route that was used. A number in your dataset is never a bare number. It is a claim with a receipt.

Records that stop appearing are marked as missing rather than deleted, so a target dropping an item from a listing page is visible as an event rather than as a row that silently evaporates.

Working with a dataset

Open a dataset from a Flow, a run, or the main navigation.

• Filter, sort, and choose fields. Save a view without changing the stored schema — a view is how you look at the data, not what the data is.
• Expand a record to see its current values, its full observation history, the source evidence, its validation status, and any related change events.
• Compare any two observations field by field.
• Export the current filter and view as JSON or CSV. Large exports run as background jobs with an expiring download rather than blocking the request.
• Page through it on the API, with the same filters. See Public API.

The JSON representation is available, but it is an alternate view, not the default. Most people working with a dataset want the table.

Retention and deletion

Datasets, records, and artifacts are subject to your organization's retention rules. Deleting records or a whole dataset is a retention-aware, confirmable job with audit history — not a button that silently drops rows.

Evidence artifacts (HTML, screenshots, traces) have their own configurable retention, separate from the records, because they are much larger and usually needed for a shorter time.