LMSYS Chatbot Arena vs Vellum in 2026

Q: Is Vellum free to use?

Vellum is free to download and start . New users get their first tasks free . If you choose Vellum Cloud, subscription plans start at $30 per month for Mighty, $100 per month for Super or $200 per month for Ultra . The Super plan includes a medium computer with 30 GB of storage .

Weighing LMSYS Chatbot Arena against Vellum: what each tool does, what it costs, and questions to ask before you choose one.

By SupaKeeper · 2026-10-20

SupaKeeper is published by LunarWerx, which makes some of the tools compared here.

LMSYS Chatbot Arena lets you compare AI models side by side ^[1], while Vellum is an assistant that can send emails and manage your calendar ^[2]. Model Odds estimates the chance of a new Anthropic or OpenAI model launch within 30 days ^[3].

Option	What its sources say it does	Cost	Install or account
LMSYS Chatbot Arena	Lets you battle 2 anonymous models and compare 2 models side by side ^[1] ^[1]	not published in the facts for this page	not published in the facts for this page
Vellum	An assistant that can send emails, manage calendars, browse the web and write code ^[2]	free to download and start ^[2]; Cloud subscriptions from $30 per month ^[2]	native to Mac and accessible on iOS, Android and more ^[2]
Model Odds	Ranks every Anthropic and OpenAI model family by estimated chance of a release within 30 days ^[3]	free, and so is its API ^[3]	a website with a public API ^[3]; a web app that needs JavaScript ^[3]

The short answer

For a developer, LMSYS Chatbot Arena is a place to test AI models against each other. It lets you battle 2 anonymous models ^[1] and compare 2 models side by side ^[1]. Vellum is an assistant that can send emails, manage your calendar, browse the web and write code ^[2]. Model Odds is a tool for timing, since it estimates the chance that Anthropic or OpenAI publishes a new model within 30 days ^[3].

Which tool meets your need depends on the need. Comparing model answers side by side points to LMSYS Chatbot Arena ^[1]. Running an assistant that handles email, calendar and recurring tasks points to Vellum ^[2]. Knowing when a new Anthropic or OpenAI model may come out points to Model Odds ^[3].

What LMSYS Chatbot Arena, Vellum is built for

LMSYS Chatbot Arena's own page says you can battle 2 anonymous models ^[1]. It lets you compare 2 models side by side ^[1] and chat directly with 1 model at a time ^[1]. It describes an Agent Mode built for complex tasks ^[1]. Its leaderboards are powered by real people doing real work on Arena ^[4]. On its leaderboard, Claude Opus 5.5 (High) is shown at number 1 with 14.33% ^[4].

Vellum's own page says it is free to download and start ^[2]. It is native to Mac and accessible on iOS, Android, the web app, voice, email, Telegram, Slack and terminal ^[2]. Vellum is open source, so you can read its code yourself ^[2]. You can switch between OpenAI, Anthropic and Gemini models, or open-weight models through Ollama, at any time ^[2]. If you use Vellum Cloud, your assistant runs in an isolated setup provisioned exclusively for you ^[2].

What Model Odds is built for

Model Odds is made by the team that publishes this site. It ranks every Anthropic and OpenAI model family by its estimated chance of a release within 30 days ^[3]. Anthropic and OpenAI each have their own page that lists every past launch ^[3]. Each past launch the model uses has to be confirmed by an official post and an independent timeline, and it is listed with its date ^[3].

The estimate uses a method fitted on the gaps between recent launches, and it takes into account how long the current wait has already lasted ^[3]. It runs three estimators side by side and shows the spread between them as an error bar ^[3]. Signals read from company SDKs and docs, and from third-party model catalogues when two of them agree, can only raise the odds ^[3]. The estimator can replay any past date without using data from after that date ^[3]. It has a public API that serves odds for each model line, the launch record and service health ^[3].

Pricing and fees

The price of LMSYS Chatbot Arena is not published in the facts for this page, so check its own pages before you rely on any cost. Vellum is free to download and start ^[2], and new users get their first tasks free ^[5]. Vellum Cloud subscriptions apply only if you choose Vellum Cloud, and they start at $30 per month for Mighty, $100 per month for Super or $200 per month for Ultra, with dedicated compute, storage and bundled credits ^[2]. The Super plan, at $100 per month, includes a medium computer with 30 GB of storage ^[5]. Model Odds is free, and so is its API, with no account needed ^[3].

Setting it up and daily use

Vellum is free to download and start ^[2]. Once it runs, it can send emails, manage your calendar, create slides, write code and run recurring tasks in the background ^[2]. It comes with built-in skills for Google Workspace, Slack, Notion and your filesystem, and you can extend it with community plugins or your own custom skills ^[2]. High-stakes actions, such as emailing new contacts or spending credits, need your explicit approval until you mark them as trusted ^[2]. The facts for this page do not describe a setup routine, so check Vellum's pages for that.

You use LMSYS Chatbot Arena to battle 2 models ^[1], compare 2 models side by side ^[1] or chat with 1 model at a time ^[1]. The facts for this page do not describe installing it, so check its pages for that. Model Odds is a web app that runs in the browser and needs JavaScript ^[3]. Its website and API share one snapshot cached for up to five minutes ^[3]. You can sign up with a confirmed email to get alerts, and every subscriber gets every company's launches at the same time ^[3].

Things to check before you choose

Ask what job the tool has to do. Comparing model answers and running an assistant that sends emails are different jobs ^[1] ^[2]. Ask where your data goes. By default Vellum keeps conversation history, memory and credentials on your machine and never uses them to train models ^[2], but if you use Vellum Cloud, the assistant runs in an isolated setup provisioned for you ^[2]. The facts for this page do not say how LMSYS Chatbot Arena handles the data you enter, so check its own pages for that.

Ask which models you need. Vellum can switch between OpenAI, Anthropic, Gemini and open-weight models through Ollama ^[2]. Model Odds ranks every Anthropic and OpenAI model family, so check whether the companies you follow are among them ^[3]. Ask whether an estimate suits your question. Model Odds shows the spread between its estimates as an error bar ^[3].

How to choose

If you need to compare model answers side by side, LMSYS Chatbot Arena matches that need ^[1]. If you want an assistant that sends emails, manages your calendar and runs recurring tasks in the background, Vellum matches that need ^[2]. If you want your conversation history and credentials to stay on your machine by default, Vellum says that is how it stores them ^[2]. If you need to know how likely a new Anthropic or OpenAI model launch is within 30 days, Model Odds is built for that ^[3]. If you want a free tool with a public API for launch odds and service health, Model Odds offers one ^[3] and it is free ^[3].

Frequently asked questions

Is Vellum free to use?

Vellum is free to download and start ^[2]. New users get their first tasks free ^[5]. If you choose Vellum Cloud, subscription plans start at $30 per month for Mighty, $100 per month for Super or $200 per month for Ultra ^[2]. The Super plan includes a medium computer with 30 GB of storage ^[5].

How much does LMSYS Chatbot Arena cost?

No price for LMSYS Chatbot Arena is published in the facts for this page, so no cost is stated here, and its own pages are the place to check. Its listed features are battling 2 anonymous models ^[1] and comparing 2 models side by side ^[1]. Check its pages for pricing before you plan any use.

Where does Vellum keep conversation history?

By default, conversation history, memory and credentials stay on your machine and are never used to train models ^[2]. If you use Vellum Cloud, the assistant runs in an isolated setup provisioned for you ^[2]. The facts for this page do not describe anything beyond that, so check Vellum's own pages before you rely on it for sensitive work.

Is Model Odds free to use?

Model Odds is free, and so is its API, with no account needed ^[3]. It is a website with a public API ^[3], and it runs as a web app in the browser that needs JavaScript ^[3]. Its website and API share one snapshot cached for up to five minutes ^[3].

How does Model Odds estimate when a new model will launch?

The estimate uses a method fitted on the gaps between recent launches, and it takes into account how long the current wait has already lasted ^[3]. It runs three estimators side by side and shows the spread between them as an error bar ^[3]. Signals from company SDKs and docs, and from third-party model catalogues when two of them agree, can only raise the odds ^[3]. Each past launch it uses must be confirmed by an official post and an independent timeline ^[3].

Model Odds is free, and so is its API, with no account needed. ^[3]

Try Model Odds