SupaKeeper Plain comparisons of AI coding tools

How to schedule comparison runs after model announcements for benchmark research teams

Plan comparison runs after model announcements for benchmark research teams, with steps, options to check, and common mistakes to avoid.

By SupaKeeper ยท

SupaKeeper is published by LunarWerx, the makers of AgentHydra, Copilot AI Productivity Suite, Codex Forecast and Model Odds, which are compared here alongside other tools.

Set up comparison runs before an announcement arrives, decide which announcements should trigger a run, and keep the test setup ready so a run can start quickly. A release tracker can help you watch for likely launches. Model Odds ranks the estimated chance that each Anthropic and OpenAI model family is released within 30 days [1], and those figures are estimates rather than confirmed dates.

OptionWhat it doesWhat to check
Model OddsEstimated chance of a release within 30 days for each Anthropic and OpenAI model family [1]Check that the estimates match the triggers on your list. Free to use, with no account needed [1]
Your team's own release watchDepends on the sources your team already reads.Check how it records dates and whether it can start a run.
Company announcement pagesThe announcement posts you already follow.Check whether each post gives a date you can plan around.

What you need before you start

Before a comparison run can start, benchmark research teams need a fixed set of things ready. Keep a written list of the models you compare against, the version string of each one, and the benchmark sets you rerun. Store the scoring scripts and the expected output format with each test set, so a new model can be run the same way as the last one.

Decide in advance which kinds of announcements matter to your team. A new model family, a new version of an existing family, and a change in access or pricing may each need a different response. Write down who starts the run, who reviews the results, and where the results are stored.

Check that you have access to each model, and that the account or key for every run is active and has enough capacity for the test calls. A run that stops because a key expired wastes the time the announcement gave you.

Keep a short log of past runs. The log shows how long a typical run took and which results changed, which helps when you plan the next one.

Step by step

  1. Pick the announcements that should trigger a run. For each one, write down the company, the model family, and the models your team would compare against.
  1. Set up a way to watch for likely releases. You can use a tracker or a regular check of the company pages your team already reads, and note the date you first see each signal.
  1. Prepare the test set and the scoring scripts before any announcement, so a run can start on the day a model appears.
  1. Decide the run order. Start with the new model, then run the baseline models it should be compared against.
  1. Freeze the settings. Use the same prompts, the same sampling settings, the same sample size, and the same test set for every model in the comparison.
  1. Record the start time, the model version string, and the environment for each run, so the results can be repeated later.
  1. Score the outputs with the same scripts used in earlier runs, and store the raw outputs next to the scores.
  1. Compare the new results with the last report. Note what changed and whether the change is likely to come from the model or from the setup.
  1. Share the report with the team and mark the run as complete.
  1. Review the trigger list after each cycle, and remove any announcement that never led to a useful run.

Tools that can do it

Model Odds is made by the team that publishes this site. Model Odds ranks every Anthropic and OpenAI model family by its estimated chance of a release within 30 days [1]. Each company has its own page that lists every past release [1]. Each past launch on the list has to be confirmed by an official post and an independent timeline, and it is listed with its date [1]. It runs three methods side by side and publishes the range between their results, so you can see how far they disagree [1]. The estimate is built from the gaps between recent launches, and it takes into account how long the current wait has already lasted [1]. Signals read from company SDKs and docs, and from third-party model catalogues when two of them agree, can only raise the odds [1]. The estimator can replay any past date without using data from after that date, so you can check how it would have behaved [1]. You can sign up with a confirmed email to get alerts, and every subscriber gets every company's drops at the same time [1]. Model Odds is a website with a public API, and the API serves per-line odds, the launch record, and service health [1]. The website and the API share one snapshot that is cached for up to five minutes [1]. It is a web app that runs in the browser and needs JavaScript [1]. Model Odds is free, and so is its API, with no account needed [1].

Other options include the release watch your team already runs, or a regular check of the company announcement pages you follow. This page has no confirmed facts about those routes, so check how each one records dates, whether it keeps a record of past launches, and whether it can start a run.

Common mistakes to avoid

Avoid starting a run before the setup is frozen. A run that changes prompts, settings, or sample size partway through cannot be compared with the baseline.

Avoid treating an estimated chance as a confirmed date. Plan the work around the window, and keep the setup ready so a late start does not waste the run.

Avoid skipping the baseline. A new score means little without the same models run under the same conditions.

Avoid letting trigger lists go stale. Review them after each cycle and drop announcements that never led to useful runs.

Avoid storing results without the version and time. Without them, a later comparison may not be repeatable.

Frequently asked questions

How do benchmark research teams schedule comparison runs after announcements?

Start by listing the announcements that should trigger a run and the models each one affects. Keep the test set, the scoring scripts, and the baseline results ready before any announcement arrives. When a release is confirmed, start the new model under the same settings as the baseline, record the version and time of the run, and score it with the same scripts. Store the raw outputs next to the scores so the comparison can be repeated later.

Is Model Odds free for benchmark research teams?

Model Odds is free, and so is its API, with no account needed [1]. It is a web app that runs in the browser and needs JavaScript [1]. Teams can check the estimated chances in the browser, or read the per-line odds through the public API [1]. Check the page before a planned run to see the current estimates.

Can Model Odds confirm the exact date of a model release?

No. Model Odds ranks the estimated chance that each model family is released within 30 days [1]. The estimate uses the gaps between recent launches and how long the current wait has already lasted [1]. Signals from company SDKs and docs can only raise the odds, never lower them [1]. Plan the run around the estimate and the official announcement, not around a single predicted day.

Does Model Odds send email alerts when new models appear?

Model Odds offers email alerts after you sign up with a confirmed email [1]. Every subscriber gets every company's drops at the same time, so the alerts reach the whole list together [1]. Treat an alert as a prompt to check the test set and the trigger list, then start the run with the same settings as the baseline.

What should be checked before using Model Odds for scheduled runs?

Check that the estimated chances cover the model families your team compares against, and that the odds match the trigger list you set. Note the snapshot age, since the website and API share one snapshot that is cached for up to five minutes [1]. Review the launch record to see the dates of past launches [1], and use the backtest to see how the estimator would have behaved on earlier dates [1].

Model Odds is free, and so is its API, with no account needed. [1]

Try Model Odds

Sources

  1. Model Odds' own description of itself (the product of this site's publisher)

Related comparisons