How to plan procurement evaluations around new AI capabilities
Plan procurement evaluations around new AI capabilities: what to prepare, the steps, options to compare, and mistakes to avoid.
SupaKeeper is published by LunarWerx, the makers of AgentHydra, Copilot AI Productivity Suite, Codex Forecast and Model Odds, which are compared here alongside other tools.
Plan the evaluation in four parts: list the capabilities you need, set how each one will be tested, pick the dates the decision must meet, and check every option against those points before committing. A timing tool can add an estimate of when new model releases may arrive, which helps decide when to schedule a test. Model Odds is one such tool. [1]
| Option | What it does | What to check |
|---|---|---|
| Model Odds | Estimates the chance of a new model family release within 30 days, for Anthropic and OpenAI. [1] | Whether Anthropic and OpenAI match the vendors you plan to evaluate, and whether its estimates fit your decision dates. |
| Your team's evaluation scorecard | A written set of criteria and test material your team already uses for software and services. | Whether it covers the capabilities and release timing questions in this plan. |
| A procurement tracker you set up yourself | A place to record vendor documents, scores, and decision dates. | Whether it can show the release timing you need, or whether that must come from another source. |
| Vendor documentation reviewed directly | The published terms, security notes, and release notes from each vendor. | Whether the documentation is current and covers each capability on your list. |
What you need before you start
Before you plan evaluations, have a clear list of the AI capabilities your organization may buy or use, such as text analysis, code generation, or document review. Write down which business tasks each capability would serve and who on the team must approve a purchase.
Gather the vendor documents you already have, including contract terms, security reviews, and any past evaluation results. Know the budget window and the dates your team must decide by, because those dates set how much time an evaluation can take.
Decide what success looks like for each capability. Include accuracy needs, data handling rules, and the people who must sign off. A plan without these points is hard to finish.
Have a rough sense of which model families your current vendors offer, so you can tell when a new release would change your options.
Step by step
- Name the capability you want to evaluate and the specific business task it must support. Write the task in plain words, such as summarizing supplier contracts.
- Set the test criteria before any vendor demo. Include accuracy goals, data handling rules, cost limits, and the people who must approve.
- Collect the vendor documents for each candidate, including terms, security notes, and data retention policies.
- Build a short test set from real past work, with known correct answers, so each option is scored on the same material.
- Set the evaluation window and the decision date. Work backward from the date a purchase must be approved.
- Plan for release timing. If a new model release could change the options, decide in advance whether to wait, test now, or retest after release.
- Score each option against the same criteria and record the reason for each score.
- Review the plan when a new capability appears, and rerun only the parts that the change affects.
Tools that can do it
Model Odds is made by the team that publishes this site.
Model Odds ranks every Anthropic and OpenAI model family by its estimated chance of a release within 30 days. [1] Each company gets its own page that lists every past release. [1] Each past launch the model uses has to be confirmed by an official post and an independent timeline, and it is listed with its date. [1] The estimate is a renewal process fitted on the gaps between recent launches, and it takes into account how long the current wait has already lasted. [1]
Model Odds runs three estimators side by side and publishes the spread between them as the error bar. [1] The estimator can replay any past date without using data from after that date. [1] Model Odds lets you sign up with a confirmed email for alerts. [1]
Model Odds is free, and so is its API, with no account needed. [1] It is a web app that runs in the browser and needs JavaScript. [1]
The other options in the table are approaches your team already has or can set up. Check each one against the criteria in the first section before choosing.
Common mistakes to avoid
Treating a release estimate as a confirmed date is a common error. Model Odds gives an estimated chance of release within 30 days, which is a planning input and not a fixed date. [1] Plan for more than one outcome instead of waiting on a single expected day.
Running tests on demo material instead of your own work tends to overstate results. Use a test set built from your real tasks, as described in the steps above.
Skipping the approval path until the end can delay a purchase after the evaluation is finished. Confirm who signs off before scoring begins.
Comparing options with different criteria makes the scores hard to trust. Use the same criteria and the same material for every candidate.
Assuming that a timing tool covers every vendor you might use is a mistake. Model Odds ranks Anthropic and OpenAI model families, so check whether your vendors fall within that set. [1]
Frequently asked questions
How do AI procurement specialists plan evaluations around new AI capabilities?
Start by naming each capability and the business task it must support, then set test criteria, build a test set from real work, and work backward from the decision date. Score every candidate against the same material and record the reasons for each score. If a new release could change your options, decide in advance whether to wait, test now, or retest after release. Writing these points down before any vendor demo keeps the evaluation fair and easier to defend when approvals are reviewed later.
Is Model Odds free for procurement planning?
Yes. Model Odds is free, and so is its API, with no account needed. [1] It is a web app that runs in the browser and needs JavaScript. [1] Its estimates work as a planning input, so pair them with your own evaluation criteria and the decision dates your team already has.
Does Model Odds give a fixed release date for new AI models?
No. Model Odds ranks every Anthropic and OpenAI model family by its estimated chance of a release within 30 days. [1] Because the output is an estimated chance and not a fixed date, it works as a timing input for planning rather than a calendar entry. Record the estimate you used alongside the date you read it, so your notes show what your decision was based on.
How does Model Odds arrive at its estimates?
Model Odds runs three estimators side by side and publishes the spread between them as the error bar. [1] The estimate is a renewal process fitted on the gaps between recent launches, and it takes into account how long the current wait has already lasted. [1] Each past launch used must be confirmed by an official post and an independent timeline. [1] The estimator can also replay any past date without using data from after that date. [1]
What should be checked before relying on release timing signals?
Check which companies the signals cover, whether the estimates match the decision dates you have, and how each signal is used. Model Odds ranks Anthropic and OpenAI model families. [1] Signals read from company SDKs and docs, and from third-party model catalogues when two of them agree, can only raise the odds. [1] Record the date you checked, so your notes show which view of the timing you used when the decision was made.
Model Odds is free, and so is its API, with no account needed. [1]
Try Model OddsSources
Related comparisons
- How to advise customers on AI model upgrades
- How AI community managers answer release date questions
- How to anticipate rate limit interruptions for API integrators
- How to audit agent commands in each session in 2026
- How to audit agent sessions across projects
- Audit What an Agent Ran in Each Session for DevOps
- How freelance coders avoid hitting limits mid-task
- How to avoid hitting the weekly cap for Claude users
- How to avoid running out mid-project in Cursor
- How to budget AI usage per sprint