In this article
Testing Jev, OpenAI and Microsoft on the same business-style decision tasks produced some clear differences. Microsoft led on average accuracy and calculated API cost, OpenAI was fastest, and Jev performed well in retail and procurement. But some wrong answers came with high confidence. The results offer a useful shortlist, while showing why the cost of mistakes and the amount of human review deserve attention when choosing a model.
I am endlessly interested in how emerging AI capabilities perform in the everyday workflows we help businesses build. Decision models are one area where understanding the trade-offs between accuracy, speed and cost can directly inform our delivery choices. I put Jev, OpenAI’s Decisions API and Microsoft-Decision-1 through the same business-style tasks to explore those trade-offs and share a practical set of findings with the wider technical community.
Accuracy gives each task equal weight. These are results for this selected sample.
But choosing a service takes more than comparing three numbers. A fast answer matters in an interactive application; a confident mistake matters when it approves something that should have gone to a person for review. The right choice depends on the work you need done.
Microsoft for accuracy and low API spending
Microsoft was the strongest candidate here when prioritising accuracy and API spending. It averaged 96.0% accuracy and scored 100% in logistics, compared with Jev’s 88% and OpenAI’s 73%.
Its typical response took about 328 milliseconds. That may be acceptable for background document checks but can matter when a workflow makes several decisions in sequence.
There were also two failed requests: a timeout and a rate limit. Neither was retried.
High confidence didn’t always mean a correct answer. Microsoft scored 95% on the insurance task but gave all five wrong answers at least 95% confidence. A rule that sent only low-confidence answers for review would have missed those mistakes.
OpenAI for speed and image inputs
OpenAI had the lowest median response time in every task, about 107 milliseconds overall, and completed all 959 requests. That makes it worth testing for routing and interactive workflows where delays accumulate.
Response times are medians, the middle result among successful test requests, rather than a guarantee for every call.
It also has a capability the others lack: OpenAI is the only one of these three services that supports native image inputs. Its Decisions documentation supports text and images; Jev and Microsoft-Decision-1 are documented as text-only.
If your workflow includes photos or scanned pages, that could materially change your shortlist. This benchmark didn’t test image accuracy, speed or cost.
The trade-off was lower average accuracy, 85.4%, and higher calculated cost. Requiring 90% confidence would let only seven of OpenAI’s 100 logistics answers through without review, leaving substantial manual work.
I couldn’t confirm OpenAI’s processing location. These timings therefore reflect the experience from my Australian connection and the service settings used. They don’t prove the model itself computes answers faster.
Jev for task-specific strengths
Jev sat between the others on average accuracy, response time and calculated cost, with no failed requests. It averaged 91.0% accuracy and a median response of about 241 milliseconds.
Retail was a strength: 95% accuracy against OpenAI’s 81%. Jev also narrowly led procurement, although its 88% versus Microsoft’s 87% is too small a gap to base a choice on.
That makes Jev worth testing for those specific workflows. Some banking mistakes carried very high reported confidence, so its review thresholds need checking against real examples too.
What the API costs showed
Across the run, the calculated costs were:
- Microsoft: about 1.66 US cents, including a conservative allowance for two failed requests.
- Jev: about 3.06 US cents.
- OpenAI: about 5.17 US cents.
Microsoft’s lower cost than Jev came from fewer reported input tokens, despite the identical listed input rate. Sending the same task to two services doesn’t necessarily produce the same billable token count.
Jev and Microsoft have the same listed input price: US$0.042 per million tokens. OpenAI’s Decisions rate is higher, at US$0.10. All three list output tokens as free. These are the base rates used in the test; OpenAI also documents regional and long-context pricing adjustments.
A token is a unit of text the provider counts for billing. I calculated successful-request cost as:
Reported input tokens ÷ 1,000,000 × listed input price.
Across the run, the arithmetic was:
- Jev: 729,685 input tokens × US$0.042 per million = about 3.06 US cents.
- OpenAI: 517,242 × US$0.10 per million = about 5.17 US cents.
- Microsoft: 385,159 × US$0.042 per million = about 1.62 US cents.
Microsoft’s total also included a conservative allowance of 0.0375 US cents for its two failed requests, bringing it to about 1.66 US cents. Actual billing for those failures is unknown.
Tokenisation, request packaging and billing conventions may differ; this experiment didn’t isolate which caused the usage gap.
These figures cover 959 attempted requests per service, including the pilot and additional checks, with no retries. They are calculations from reported usage and listed prices, not verified invoices, and exclude tax, currency conversion and earlier preparation.
How decision models fit into a business workflow
A decision model takes information, instructions and possible answers, then scores those answers. Think “approve, reject or review” for a refund, or choosing which team should handle a customer message.
Software can use those scores to route work, with people checking the uncertain cases. The findings above show why both accuracy and confidence need testing: a model can give a wrong answer with high confidence, while a cautious model can leave substantial work for people to review.
What went into the test
The live test ran from Australia on 10 October 2026, using Jev 1.13.0, gpt-6-luna through OpenAI Decisions, and Microsoft-Decision-1 version 1. Each service received the same evidence, instructions and answer options through its own API.
There were eight sets of 100 cases:
- Banking: sort customer-support messages into 77 possible categories.
- Retail: approve, reject or review a refund.
- Logistics: choose the next step for a parcel.
- Manufacturing: grade a defect’s severity and check whether there was enough evidence.
- Security: allow, block or review a proposed tool action.
- Healthcare administration: check whether a document packet was complete.
- Insurance operations: decide whether documents supported or contradicted a statement, or left it unresolved.
- Procurement: score how well four candidates matched requirements.
Banking used original labelled messages from PolyAI’s BANKING77 dataset. The other seven sets were built from synthetic records and explicit fictional rules. Each contained 60 ordinary cases, 30 boundary cases and 10 with insufficient or conflicting evidence.
A few examples show what the models had to handle:
- A refund could fall exactly on a policy’s time limit, with an exception taking priority over the usual rule.
- A parcel log could contain old events or unrelated archive records. The answer had to follow the latest valid evidence for the right parcel before the cut-off.
- An insurance statement needed to be checked against receipts and estimates. Missing evidence had to remain “insufficient”, rather than being treated as proof that the statement was false.
Separate rule-checking programs verified the expected synthetic answers. Checks also confirmed that converting records into text preserved the facts.
The test cases and scoring rules were fixed before collecting model predictions and weren’t retuned after seeing the results. No paid model supplied the reference answers; there was no independent industry-expert review.
These were controlled text exercises. Healthcare involved paperwork, not diagnosis; security actions were never executed. Ten manufacturing cases had unknown severity and were excluded from that primary score.
Where I would start
For these text tasks, Microsoft is the strongest starting point for accuracy and API cost, OpenAI for response speed, and Jev deserves a closer look for retail and procurement. Image input gives OpenAI a separate reason to be considered.
Before picking any model, check whether ordinary code will do. Rule-based code scored 100% on the seven synthetic sets without API calls because the facts were structured and the rules explicit.
The next useful test is a representative sample from your actual workflow, with independently checked answers. Compare costly mistakes and the amount of human review each service needs alongside speed and usage. One short run can help build a shortlist; it can’t establish a universal winner.
I’m interested in alternative views and findings from similar tests or production deployments, particularly where different data, rules or review thresholds have produced different results. Sharing those perspectives helps build a fuller understanding of where these models can add value.
