SupaKeeper Plain comparisons of AI coding tools

How to compare release timing across AI labs for model evaluation teams

A tool-neutral guide for model evaluation teams comparing AI release timing, with Model Odds as one option to check.

By SupaKeeper ยท

SupaKeeper is published by LunarWerx, the makers of AgentHydra, Copilot AI Productivity Suite, Codex Forecast and Model Odds, which are compared here alongside other tools.

To compare release timing across AI labs, list the labs and model families you follow, record each past launch with the source that confirms it, and check estimated chances against official announcements. Model Odds ranks Anthropic and OpenAI model families by their estimated chance of a release within 30 days. [1] Treat that figure as a likelihood, and check it against each lab's official announcements before planning a test run.

OptionWhat it doesWhat to check
Model OddsEstimated chance of a release within 30 days for each Anthropic and OpenAI model family [1]Whether the labs and time window you need are covered. Free to use, with no account needed. [1]
Spreadsheet you keep yourselfA dated record filled in from each lab's own announcements.How much upkeep it takes, and how each date is confirmed.
Another trackerA separate tool, if one suits your workflow.Which labs it follows and what source each date rests on.

What you need before you start

Model evaluation teams need a short list of the model families they track and the labs that release them. Write down which releases would change an evaluation schedule, because a new model that arrives in the middle of a test cycle can alter what has already been planned.

Keep a dated record of past releases close at hand. Past gaps between launches give a baseline for judging what may come next. Each date should note its source, such as an official post or a secondhand report, because mixing the two without a label causes confusion later.

Decide which question you are answering. A ranking of likely releases helps with planning, while a confirmed date is needed before committing a test run, and the two come from different kinds of source. Model Odds works in estimated chances, which helps with ranking but does not replace an announcement. [1]

Finally, pick one place to record what you find, such as a shared spreadsheet, so the comparison stays consistent as new launches appear.

Step by step

  1. List the labs and model families you want to compare, and write down the release history you already trust.
  2. Mark each past launch with the source that confirms it, so unconfirmed dates stand apart from confirmed ones.
  3. Decide the time window that matters for your planning, such as the next few weeks or the next quarter.
  4. Collect the estimates you plan to compare, and note the date each one was made.
  5. Put every lab on the same footing by using the same window and the same kind of estimate for each.
  6. Check each estimate against official announcements as they arrive, and record whether it held.
  7. Review the record after each release cycle to see which sources were closest and where the gaps were.
  8. Keep the method the same from one review to the next, so changes in your results come from the labs and not from your process.

Each step works the same whichever tool, if any, you use. A comparison is only as good as the dates behind it, so check each one before drawing a conclusion.

Tools that can do it

Model Odds is made by the team that publishes this site.

Model Odds is a website with a public API. [1] A public API is a set of web addresses that other programs can call to fetch the same data the website shows. [1] It is a web app that runs in the browser and needs JavaScript. [1] That means it opens in a browser tab, and the page will not load if JavaScript is turned off.

Anthropic and OpenAI each get their own page that lists every past release. [1] Each past launch it uses has to be confirmed by an official post and an independent timeline, and it is listed with its date. [1] The same confirmation rule applies to every past launch.

Model Odds runs three estimators and publishes the spread between them, so you can see how far the estimates disagree. [1] The estimate is a method fitted on the gaps between recent launches, and it takes into account how long the current wait has already lasted. [1]

The estimator can replay any past date without using data from after that date, which lets you test how it would have done. [1] You can sign up with a confirmed email to get alerts, and every subscriber gets every company's drops at the same time. [1] The website and the API share one snapshot that is cached for up to five minutes. [1] Model Odds is free, and so is its API, with no account needed. [1]

Other options are a spreadsheet you keep yourself from each lab's own announcements, or a separate tracker. Compare each one on the same time window and on how it confirms its dates.

Common mistakes to avoid

Treating an estimated chance as a confirmed date is the most common error. A ranking shows likelihood, and planning decisions should wait for an official post. When the three separate estimates disagree widely, hold conclusions loosely. [1]

Comparing labs over different windows gives misleading results. Use one window for every lab before reading the figures side by side.

Leaving out unconfirmed dates hides how often predictions missed. Keep them in the record with a label instead of deleting them.

Relying on a single signal misses changes. Model Odds reads signals from company software development kits (SDKs), the add-on packages companies publish, and from third-party model catalogues when two of them agree. Those signals can only raise the odds. [1] Check whether a rise came from an official source before acting on it.

Skipping a test against past dates is another error. Replaying earlier dates shows how an estimator would have behaved before you rely on it. [1]

Frequently asked questions

How can model evaluation teams compare release timing across AI labs?

Start by listing the labs and model families you follow, then record each past launch with the source that confirms it. Choose one time window for every lab so the comparison is fair. Check estimates against official posts as new releases appear, and note which sources were closest over time. Keep the method steady from one review to the next, so differences reflect the labs rather than your process.

Is Model Odds free for release timing work?

Model Odds is free, and so is its API, with no account needed. [1] It is a web app that runs in the browser and needs JavaScript. [1] Its figures are estimated chances, so compare them with official announcements before planning a test run.

Does Model Odds show which past launches are confirmed?

Each past launch it uses has to be confirmed by an official post and an independent timeline, and it is listed with its date. [1] Anthropic and OpenAI each get their own page that lists every past release. [1] Together these give a dated record to compare against your own notes.

Can Model Odds send release alerts to a whole team?

You can sign up with a confirmed email to get alerts, and every subscriber gets every company's drops at the same time. [1] The notices arrive at the same time for every subscriber. [1] Model Odds covers Anthropic and OpenAI model families, so check that these match the labs your team follows. [1]

How current is the data in Model Odds?

The website and the API share one snapshot that is cached for up to five minutes. [1] Caching means the stored copy of the data is reused for a short period instead of being rebuilt on every request, so a figure can be up to five minutes old. A figure can therefore lag behind a release by that window, so note when you pulled each number. [1] Keeping a dated log of readings makes later comparisons easier to explain.

Model Odds is free, and so is its API, with no account needed. [1]

Try Model Odds

Sources

  1. Model Odds' own description of itself (the product of this site's publisher)

Related comparisons