Introducing Ray9

The stack nobody meant to build

Ask anyone who collects web data for a living what their setup looks like, and you will hear a list rather than a product.

There is a browser automation library, because some pages need JavaScript. There is a proxy vendor, because some targets block cloud IPs. There is a hosted scraping API, because the proxy vendor did not solve the hard target. There is parsing code, because the API returns markdown and what you needed was a table. There is a scheduler, because the data has to be fresh. There is a database, because the data has to persist. There is a webhook system, because the data has to go somewhere. And there is a person, usually one, who understands how the pieces fit together.

Nobody set out to build that. It accumulated. Each piece was a reasonable answer to a problem the previous piece created, and each one is genuinely good at what it does. But the seams between them are where the work actually lives, and no vendor owns a seam.

So the failures land in the gaps. A page changes its markup and the parser starts returning empty strings — but the scraping API returned 200, so the scheduler thinks it worked, so the webhook fires, so the database fills up with rows of nothing. Three weeks later somebody notices that a competitor's prices have been flat since April.

That is not a bug in any one of those tools. Every one of them did exactly what it was asked. The problem is that nothing was watching the whole thing.

What is actually missing

The gap is not a better scraper. There are good scrapers. The gap is that the lifecycle has no owner.

A working web data pipeline has to do six things, and they have to know about each other:

Build. Turn an intention — "every product on this category page, with price and availability" — into a definition. Today this is a person writing selectors, which is fine until the person leaves.

Run. Get the page. Which is not one problem but four: reaching the target, rendering it, not being blocked, and doing all that at a cost that makes sense for the value of the data.

Validate. Establish that what came back is what you asked for. This is the step almost everyone skips, and it is the one that quietly poisons everything downstream. HTTP 200 is not success. A challenge page returns 200. A consent wall returns 200. A login screen returns 200. The wrong country's version of the site returns 200 with entirely plausible, entirely wrong prices.

Store. Keep the records with a stable identity, so that today's observation of a product can be compared with last week's rather than piled next to it.

Monitor. Notice when something meaningful changed — which requires knowing what "meaningful" means, and which is why raw HTML diffing produces alerts that everyone learns to ignore within a fortnight.

Deliver. Get the records somewhere useful, exactly once, without losing them when the receiving end is down.

Each of these is solvable. Several of them are solved. What is missing is a single system that holds all six, so that the validation step knows what the build step promised, and the monitoring step knows what the storage step considers to be the same record, and the whole thing can tell you afterwards where a particular number came from.

The thesis

Ray9 is that system. One product, owning the lifecycle.

You describe what you want and give a URL. Ray9 opens the page, proposes a schema with typed fields, and shows you the exact region of the page behind every one of them — evidence rather than assertion. You correct what it got wrong by clicking on the page.

Then you publish. Publishing creates an immutable version, and the separation between a draft you can freely break and a version that is running in production is the load-bearing idea of the whole product. Every run points at an exact version. Every record points at the run that produced it. Six months later, a number in your dataset can be traced to the precise definition, the precise page, and the precise capture that produced it.

Runs go out on an execution route, chosen from what the work actually requires: a direct HTTP fetch where the page is simple, a managed provider where that is the reliable path, an isolated cloud browser where JavaScript or interaction or anti-bot behaviour demands one. Egress — datacenter, residential, mobile — is chosen separately, because IP reputation and browser fingerprint are different problems that fail for different reasons. Official and licensed connectors have a reserved place in the same model, ranked above page scraping wherever one exists; building them out is ahead of us rather than behind us.

Cheapest first, but only among the routes that can actually satisfy the request. If a job requires a specific country, an authenticated session, or a headful browser, those are hard constraints. Ray9 will not quietly drop one of them to make a run succeed on a cheaper route, because a run that returns the wrong country's prices with a green tick beside it is worse than a run that failed.

Results are validated on their own terms — transport, page correctness, extraction, schema, evidence — and a run ends as complete, partial, blocked, action required, failed, or cancelled. Never as a boolean.

Records land in a dataset with a current view and an append-only observation history, so "what did this cost last Tuesday" is a question with an answer. Monitoring compares normalized values rather than raw HTML, suppresses known noise, and can require a change to be confirmed across several checks before it says anything — because a stock counter that flickers is not news. Delivery goes out as signed webhooks with idempotent event IDs, bounded retries, and a dead-letter state, so a downstream outage is a delay rather than a data loss.

And all of it is metered in one unit. A credit range is estimated before a run, a maximum is reserved before dispatch, and every grant, reservation, charge, and refund is an entry in an append-only ledger. A crawl that discovers ten times more pages than expected stops at a clean boundary rather than spending ten times the money.

Things we decided not to pretend

Some deliberate omissions, stated plainly, because a product's honesty is easiest to judge by what it declines to claim.

Ray9 does not reach every website. Nothing does. Some targets block cloud infrastructure comprehensively; some should be reached through an official API instead of a page at all. Ray9 tells you which, and why, rather than retrying quietly until your credits run out.

Ray9 does not bypass logins, paywalls, or access controls. Defaults favour public, logged-out, non-sensitive data, and policy is evaluated before any credits are spent. Authenticated collection is possible where you have the right to it and supply your own credentials — with isolated scopes, encryption, short retention, and an audit trail. auth_required is a statement of fact, not an obstacle for a sufficiently clever prompt. You remain responsible for what you collect and how you use it, and none of this is legal advice.

Ray9 does not learn its way out of thin evidence. Route selection today is a deterministic, classified waterfall. Adaptive routing — learning which route works best for a target class — is enabled only where there is enough validated evidence to justify it. A router that learns from too little data is a router that is confidently wrong, which is the worst thing a router can be.

Local execution is not shipped. Running a Flow in your own browser, on your own device, with your own session, is on the roadmap, and the product is designed around an explicit trust boundary for it. It is not available today, and we would rather say so than let a roadmap item sit unmarked on a feature list.

Where this goes

The same domain model is reachable from the dashboard, the public API, the CLI, and an MCP server. Not a UI with an API bolted on afterwards, but one set of application services with several front doors, all observing the same authorization, idempotency, policy, credit, and audit rules. What you can do by clicking, an agent can do under a scoped key — and it will be subject to exactly the same denials.

That symmetry is the part we think matters most as more of this work gets handed to agents. An agent that can spend money on your behalf needs to hit the same budget ceilings, get the same classified failures, and leave the same audit trail as a person would. Giving an agent a tool that blocks for four minutes and returns either data or an exception is how you get agents that time out, retry blindly, and spend your credits twice.

Ray9 is in early access. Pricing is provisional, the template catalog is small and maintained by us, and there is a great deal still to build. But the shape is right, and the shape is the thing that is hard to change later.

Describe the data. We will show you the evidence before you spend a credit.