How much does a correct answer cost?

Many suppliers publish prices, features and reviews of their own models. We can gather this information with source and date, without making paid calls. The price list on this page comes from the OpenRouter public catalog. The results of the 288 calls come from the technical test of the Rovemark and are presented separately. Third party results must be attributed to the authors, without being presented as our validation.

When a concrete decision depends on something that the available sources do not answer, such as compliance with a format in our instructions or the cost of a specific flow. We first define the question and check existing studies. If evidence is still lacking, we propose the smallest appropriate test, with criteria and budget defined before execution. A test with synthetic tasks does not replace research on productivity in companies.

It compares six models in this set of 24 tasks in Portuguese, under the registered routes and configurations. A higher rate here does not prove overall superiority. We don't test all possible writing, open-ended thinking, or workflows.

The observed cost helps choose candidates for similar tasks. The suite does not test actual tool calls, long documents, images, private data, or all of a product's security controls. A model change in production needs evaluation of the flow in which it will be used.

Price per token does not tell you how many tokens the model will consume or how many responses will be usable. We divide the expense of all attempts by the number of those that passed the criterion. Wrong answers also cost and are included in the total.

These are 24 tasks, each repeated twice. Repetitions help observe variation, but do not expand diversity to 48 independent problems. The results are descriptive; We do not publish confidence intervals or statistical significance for a population of tasks.

When querying the endpoints, four of the six routes did not declare support for the temperature parameter. In them, we omit the adjustment and record the provider's default. In the other two, we use zero. The exit limit was 700 tokens for everyone. Temperature and reasoning effort of different families are not assumed to be equivalent.

The inspection section shows the model's request, reference, and two responses. Approval requires the same values ​​and keys in the JSON object; the order of the keys and any possible code fence JSON are ignored. Complete data preserves failed responses and API failures.

The exact run files, protocol, reference responses, sort seed, and results are available for download. A new run requires its own credentials and budget. Availability, pricing, and provider behavior may change, even while maintaining the model identifier.

The inference cost is just one part of the operation. Fees, taxes, exchange rates, infrastructure and human review are excluded from this indicator. The test does not measure company productivity. The separate production and review protocol with companies continues without its own published results.