# Adding a System One Model: Where a Typed Decision Beats an LLM AI Data Stream now runs two kinds of model. Frontier LLMs still do the talking; TypeSafe AI's Jev makes small, calibrated decisions in under a second. What a System One model is, how it differs from an LLM, and how we used it to fix GA4's Unassigned bucket. AI Data Stream has always run on frontier language models: Claude, GPT, Gemini and DeepSeek, on your own API keys. They write the analysis, run the tools, and hold the conversation. For that work there is nothing better. But an analytics product is full of small decisions that are not conversations. Is this referrer an AI assistant? Is this stage the closed-won one? Is this search result relevant to the question? Every time we reached for an LLM to make one of those calls, we paid for a full text generation, waited seconds for it, parsed prose or JSON out of the reply, and got back a verbalised confidence that meant very little. Every time we did not, we wrote a keyword list and hoped. This week we added a second kind of model to the app. This post is about what it is, how it differs from the models you already use here, and the first place it shipped. ## What a System One Model Is [TypeSafe AI](https://typesafe.ai/) calls its models "System One", after the fast, intuitive mode of thinking in Kahneman's framing, as opposed to slow, deliberate reasoning. Their first model is **Jev**, and its contract is unlike any LLM API: - You send a **state**: a string, or a JSON object describing the situation. - You send a map of **typed questions**. There are three types. A **Choice** picks one option from a list you define. A **Noul** answers yes or no. A **Score** places the state on ordered levels you describe. - You get back **typed answers with probabilities**. A Choice returns the full distribution across your options plus a confidence. A Noul returns the probability of yes. Nothing else. The model never generates text. A request looks like this: ```json { "state": { "source": "perplexity", "medium": "(not set)", "ga4_channel": "Unassigned" }, "model": "jev-latest", "questions": { "channel": { "type": "choice", "instructions": "Which acquisition channel does this session source really belong to?", "criteria": { "ai_assistant": "An AI assistant, chatbot or AI search product", "organic_search": "A web search engine result", "self_referral": "The business's own domains or hosted pages", "link_scanner_or_tool": "A link scanner, widget CDN or collaboration tool, not a human" } }, "is_ai": { "type": "noul", "instructions": "Is this source an AI product?" } } } ``` And the answer: ```json { "answers": { "channel": { "type": "choice", "choice": "ai_assistant", "probabilities": { "ai_assistant": 0.99, "organic_search": 0.01, "self_referral": 0.0, "link_scanner_or_tool": 0.0 }, "confidence": 0.99 }, "is_ai": { "type": "noul", "noul": 0.96 } } } ``` ## How That Differs From an LLM Four differences matter in practice. **Output type.** An LLM returns a string. Even in structured-output mode it returns a string that happens to parse as JSON, and it can still put a value in the enum that you did not define, or a number with the wrong sign. Jev returns a value from the set you gave it, or a probability. There is no parsing step and no invalid-output path. That sounds small until you have written the retry logic for the other kind. **Sampling.** An LLM produces its answer one token at a time, so a twenty-row classification means twenty sequential generations or one long one. Jev evaluates every question in a request in parallel over the shared state. We send twenty rows and forty questions in one call and get forty answers back together, in about a second. **Confidence.** Ask an LLM how sure it is and it writes a number that reads well. Jev is trained with reinforcement learning for calibrated decisions, so its confidence is a statistic you can threshold on. In our testing, the answers it got wrong all came back with confidence under 0.5, and the answers it got right came back above 0.9. That gap is the whole product. **Cost and speed.** TypeSafe's advertised early-access price is $0.042 per million input tokens, with no charge for output. New accounts start with $5 of credit, and after all our building and testing across several projects the balance still reads $4.98. At that rate classifying 156 traffic sources works out at less than half a cent. End-to-end latency from our servers has been 650 ms to 1.2 s per request. At that price you stop asking whether a decision is worth a model call. What it gives up is everything an LLM is for. It cannot explain its answer, write a summary, or hold a conversation. Code has to own the workflow and pose narrow questions; the model supplies judgement where ordinary code would need a keyword list. ## The First Place We Used It: GA4's Unassigned Bucket Google Analytics 4 groups traffic into default channels with rules over the source and medium strings. The rules are decent, and this year they gained an **AI Assistant** channel. But a rule only fires when the medium tag it expects is present. On one of our own sites, superyachtwatch.com, GA4 credited 510 sessions to AI Assistant in the last 90 days, and left another 23 in **Unassigned** as `perplexity / (not set)`, `copilot.com / (not set)` and an untagged `chatgpt.com / (none)`. Across the properties we tested, Referral held the company's own subdomains and HubSpot preview pages, Microsoft Teams link previews, Salesforce pages, link-in-bio tools and referrer spam, all counted as real external visitors. Fixing that with rules means maintaining a list of every AI product, every scanner and every self-referral pattern for every customer. Fixing it with an LLM means a multi-second generation on every traffic query. It is exactly the shape of problem a System One model is for. **The questions.** For each row GA4 filed under Referral or Unassigned, we ask a Choice over a sixteen-option taxonomy (AI assistant, self-referral, link scanner or tool, referrer spam, industry referral, other referral, the standard search, social, video, email and display channels, offline or campaign tag, unknown) and a Noul: is this source an AI product? The state carries a short description of the property, because "self-referral" depends on knowing whose site it is, plus the rows themselves. Twenty rows go in one request, referenced by path in each question's instructions. **The validation.** Before wiring it in we ran 156 real source and medium rows from two properties through it. It agreed with GA4 on 116. Of the 40 disagreements, almost every one was GA4 giving up: AI assistants in Unassigned, own domains and link scanners in Referral, Brave Search and ya.ru in Referral, a webmail host in Referral, QR codes and campaign labels in Unassigned. Three answers were wrong. All three reported confidence below 0.5. **The design rules that came out of that.** 1. Only Referral and Unassigned rows are sent. `google / organic` does not need a second opinion. 2. Below 0.7 confidence, the GA4 grouping stays and the model's answer is attached as a suggestion only. GA4 remains the source of truth for the row. 3. Every answer is cached for 30 days per property, source and medium. Each host is judged once; the extra second applies to the first lookup only. 4. A model failure, timeout or rate limit is a null result, never an error. The row comes back exactly as GA4 returned it. Early-access APIs change their limits without notice, and we would rather degrade silently than fail a traffic query. 5. When the taxonomy changes, the cache key version changes with it, so old answers are re-judged against the new options. We already did this once, to add `referrer_spam` after a crawler-service domain landed in `other_referral`. The result is a `channel`, an `is_ai_assistant` flag and a `channel_confidence` on the rows that needed them, and a status block that says whether the classification ran. Your AI can total the flag and give you an honest AI-traffic number. ## Engineering Notes A few things we learned that are not in the docs. **Candidates go in the criteria, evidence goes in the state.** The model cannot choose an option you did not offer, so for anything like "which of these existing records does this match" the records themselves are the Choice options and a `none` option is mandatory. The full text lives in the state, keyed so the instructions can point at it. **Ask independent questions together.** Questions in one request cannot see each other's answers, but they share the state and run in parallel. Speculative questions are cheap: ask the branch-specific question up front and ignore the answer if the branch is not taken. **Confidence is about the distribution, not about the world.** A confident wrong answer is possible when the right option is missing from the criteria. Calibration tells you the model is sure among the options it was given. Coverage is your job. **It is a platform key, not BYOK.** Unlike the LLMs in AI Data Stream, this model runs on our key, not yours. At the current price that is the right call, but it is also why the feature is an opt-in beta rather than a default: the terms are early-access, and we want to see real usage before we lean on it harder. ## What Comes Next Once you have a cheap, fast, calibrated judgement, you start seeing places for it everywhere. The ones on our list are reranking knowledge-base search without an embedding dependency, checking a scheduled report actually answers its brief before it is emailed, and deciding whether a new team memory duplicates an existing one. Each is a one-question integration with the same fallback rule. The traffic classification is live now as an experimental beta on **Pro and Business plans**. Team owners can switch it on in **Team Settings → Beta Features**, compare a traffic-sources query with it on and off, and [tell us](/contact/) where it gets a source wrong. TypeSafe's own documentation is at [docs.typesafe.ai](https://docs.typesafe.ai/) if you want to try the model directly.