Season 0.5. The designer agreement read is on Nov 6.

How Rams measures design

Rams puts one number on how well an interface is designed. This page is the method behind that number: what we look at, how it is computed, how sure we are, and where it can be wrong. It is our opinion, held to a fixed method and checked against designers.

Method: 1.1
Updated: Oct 16
Judge: Claude Sonnet 4.6
Cross-judge: GPT-5.6 Terra
Viewports: 1440 · 390 px
Rules: 356

Designer check pending · read Nov 6

Three measures

One judge and one method, applied to three kinds of work.

Rams Index
Models
Every model builds the same pages from the same written briefs. We render each page and grade it. Scored 0 to 100.
Rams Index · the web
Live sites
The public home pages of well-known products, rendered as a signed-out visitor sees them. The current edition ranks 132 sites on the same scale as the models.
Rams Score
Pull requests
The UI code in a pull request, read against 356 public rules. Scored 0 to 100 and posted on the pull request with fixes.

What we look at

Rendered pages for the two indexes. Code for the pull request score.

Seven grades: Hierarchy, typography, spacing, color, components, composure and craft. The judge grades each from 1 to 5, once on the desktop render and once on the phone render.
Two widths: Every page is rendered in Chrome at 1440 px and at 390 px, and both screenshots are judged. A page that only works at one width shows it.
Rendered checks: Measured, not judged: text contrast, text under 12 px, tap targets too small for a finger, content wider than the phone, and overlapping or clipped elements. The same page gives the same result every time.
The rules: 356 public rules across 9 categories, from accessibility to motion. The pull request score reads code against them. Every rule is listed on the rules page.

How the number is computed

Plain arithmetic on the judge's grades and the review's findings.

Rams Index

(grade sum − 7) / 28 × 100

The grade sum runs from 7 (every grade a 1) to 35 (every grade a 5). We average desktop and phone, then the judge's passes, then the briefs, so every brief counts the same. The rescale is a straight line, so ranks and ties are the grade sum's own.

The web

The same formula on the same seven grades, so a site and a model sit on one scale. The first cut is one judge pass per site. Beside it we print the page score, the number a Rams page review returns.

Broken pages

A build is broken when it comes back with no page, because the token budget went to reasoning, or with a page cut off before it finishes. A broken page is counted on the model’s row and never scored. A model is ranked only with a scored page on every brief, so it is never ranked on the briefs it happened to finish. One without is listed as unranked, with the reason.

Defects stay separate

On the Rams Index the rendered checks are counted beside the score and never blended into it. A page can look great and still ship small text.

Rams Score

Each issue in the code carries a severity, and the score deducts points by severity. Critical issues cap it: one at 59, two at 49, three or more at 39. A 60 or above always means no critical issue.

How sure we are

Every Index score carries an interval, and ranks come from the intervals.

Intervals: Each model gets a 95% interval. We resample the briefs, the builds of each brief and the judge passes on each page 4,000 times, paired across models so every model faces the same draws.
Ranks: Rows are numbered by Rams Index, 1 for the highest. Clearly better means the 95% interval on the paired difference sits above zero. Under each number is its rank range: every place the paired comparisons allow, from 1 plus the models clearly better to the total minus the models clearly worse. A wide range means the board cannot order that model yet, and says so.

Judges

A model grades the pages, so we check the model.

Primary: Claude Sonnet 4.6 grades every page. On 37 real home pages and 21 iPhone apps we also tried Claude Opus 5.5, GPT-6.1 Sol and OpenAI’s Decisions API as judges. None did better on both the web pages and the app screenshots, so the judge stayed.
Cross-judge: GPT-5.6 Terra judges the same pages through the same steps, and its score is printed beside ours. A Claude judge ranks Claude models, so we measure how much kinder it is to Claude pages than the second judge is. On the current board that difference is −0.5 points (95% interval −4.0 to +3.2): no sign that either judge favors its own lab’s models.
Next: Both judges’ orders are printed on the Index. Next is an open-weight judge calibrated only on designer ratings, so the measure belongs to no lab. Planned, not built.

Repeatability

Run it twice, get the same answer. ICC is how closely two judge passes on the same page agree, where 1 means identical.

Real home pages

0.97

Grade sum ICC across two passes on 37 sites.

App Store screenshots

0.94

Grade sum ICC across three passes on 21 apps.

Generated pages

0.76

Grade sum ICC on 15 model-built pages judged twice. They sit close together, so passes swap them more often.

Pull request score

Average move in points when the same review runs again, from 96 reruns on real pull requests. About 9 in 10 land within 2 points.

Agreement with designers

The test that decides whether the number means anything.

Status: Pending. Outside designers, paid and working alone, compare pairs of pages without seeing any score: 37 home pages, 21 iPhone apps and 45 model-built pages.
The rule: Written down on Oct 6 and amended once on Oct 7, before any answer came back. Rams passes on a surface if its agreement with each designer is clearly above zero and within 0.10 of how well the designers agree with each other (Kendall tau). If the designers barely agree with each other, the read is void.
When: The read is Nov 6. We publish the result whether it passes or fails, with the numbers.
So far: Against one person’s quality tiers for the 37 home pages (design-led, typical and rough, set before scoring), the grade sum agrees at Kendall tau 0.71. One rater is a sanity check, not validation.

Keeping the test fair

A number people aim at gets gamed. These are the defenses.

Private briefs: From season 1, 25 private briefs carry the rank. They are never published, and pages built from them are never shown. The current board uses the 5 public briefs only.
Anchors: The 5 public briefs, on rams.ai since July, run beside the private set. A model that does better on them than its private score predicts, by more than the field does, gets a note on its row.
Rotation: Each quarter the five oldest private briefs retire into the public sample, and five new ones replace them.
Leak checks: Before every run and on the first of each month, we search for each brief online. A brief found word for word is retired the same day, the board is recomputed without it, and the changelog says so.
We run every model: At the provider’s default settings: we send no effort or thinking setting, and where the default is known the row names it. Each model runs through its lab’s own API or, since season 0.5, through OpenRouter pinned to one serving provider (the lab itself where it serves the model), and each model’s page names the route and the provider that served its pages. Rams pays for every run. Labs don’t send outputs, don’t see results early, can’t pick their best page, and can’t pay to be listed, run again or removed. If a listed lab is a Rams customer, its row says so.

What we don't do

Lines the measure never crosses.

Customer code: Never used to build or tune the measure. Code sent for review is analyzed and discarded, never stored and never used for training. Details are on the privacy page.
Screenshots: We don’t publish screenshots of other companies’ sites, and we don’t license or sell them as training data.
Blocks: We don’t get around robots.txt, logins or bot checks. A site that blocks us is skipped and named, never scored.
The judge’s prompts: Stay private, with the rubric wording and the check thresholds. The method, the scores and the intervals are public.

Known weaknesses

Where the number can be wrong today, and what we do about it.

Brand halo: The judge often knows whose page it is: it named the site in 45 of 140 summaries in the first web cut. So we tested it. On 25 sites we hid the name and logo and judged again, beside a control that made the same edit to ordinary words. Hiding the brand cost no more than the control (+0.5 points, 95% interval −2.9 to +4.3), so we see no brand halo bigger than about 4 points. We repeat the test each season.
Banners: Cookie banners and sign-up sheets cover the page. Since 10/10 the renderer hides consent banners from 28 consent platforms and floating cookie, region and sign-up prompts it recognizes by their words, without clicking anything, and the row says what it hid. Prompts that sit in the page flow stay, because a visitor sees them too.
Bot walls: A challenge page can be graded as if it were the site. Since 10/10 the renderer refuses bot walls, access-denied and maintenance pages before the judge runs (it caught all five that got past the old check). A person still looks at every screenshot before an edition goes out.
Generated pages: Frontier models all build competent pages, and the scores sit close together. The judge repeats less well on them (ICC 0.76) than on real sites, so small gaps between models are noise: every place on the board carries its rank range, all the places the paired intervals allow.
Our renderer: Pages that need WebGL can fail in headless Chrome. A page that didn’t render is withdrawn, not scored.

Corrections

How a site, a lab or a team can dispute a score.

Tell us: If you think a site, a model or a pull request is scored wrong, email rams@rams.ai with the page and what looks off. A person reads every one.
What we do: We render the page again and judge it on the current method. If our render was wrong (a bot wall, a banner we should have hidden, a page that didn’t load), we fix or withdraw the score.
If we were right: The score stands and we tell you why. Nobody can pay to change a score. Any score we change goes in the changelog with the reason.

Versions

A score never moves without a line in a changelog.

Boards: Every board names its method version, judge model and engine commit. When the judge, the renderer or the checks change, we judge every saved page again on the new version, bump the version and keep the old board with its date.
Pull request scores: Each review names the engine that produced it. Rule changes are logged in the rules changelog.
This page: 2026-10-16Method 1.1Season 0.5: seven more labs, run through OpenRouter with the serving provider recorded, all at provider defaults. A build that comes back with no page or cut off before it finishes is broken: counted, never scored. A model without a scored page on every brief is shown unranked.
2026-10-10Method 1Published with the Rams Index (season 0): the cross-judge result, the brand-halo test, and the renderer’s bot-wall and banner handling.
2026-10-09Method 1First version of this page, in draft.
2026-10-07Index 0The Rams Index headline became the grade sum rescaled to 0 to 100. The finding-based visual score moved to a secondary column.

Questions

The short answers people ask for most.

What is the Rams Index?

A leaderboard of how well models design interfaces. Each model builds the same pages from the same written briefs. Rams renders every page at 1440 and 390 px and grades it on seven dimensions a designer would look at. The Index is the sum of those grades, rescaled to 0–100.

Why the grade sum and not one overall score?

Seven separate grades repeat better than a single verdict. On real webpages the grade sum separates design-led, typical and rough sites more clearly than our finding-based visual score, and two passes on the same page agree more often. The visual score is still shown, as a secondary column.

How is rank decided?

Rows are numbered by Rams Index, 1 for the highest. A model is clearly better than another when the paired 95% interval on the difference sits above zero. Each row also shows its rank range: every place those comparisons allow, from 1 plus the models clearly better to the total minus the models clearly worse. A wide range is a finding, not an omission: the board cannot order that model yet.

Does a Claude judge favor Claude models?

We run the same pages through GPT-5.6 Terra as a second judge and print it beside the score. No sign the Claude judge favors Claude models. The Claude judge puts the Claude-built models 4.8 Index points behind the OpenAI-built ones. Terra puts them 4.3 Index points behind, on the same scale. The difference is -0.5 points (95% interval -4.0 to +3.2), so a lean of up to 4.0 points either way is not ruled out yet. The Judges section of the Index shows each model's order under both. OpenAI Decisions (planned, v4) is the third judge from v4.

Who checks the judge?

Designers. This is season 0.5, published before their read: working designers rank the same pages blind, and the read rule was fixed before any of their answers were opened. Until that read (2026-11-06), this is the judge's opinion: stable and repeatable, not yet validated by designers.

What is private?

The brief texts of the season-1 set and the judge prompt. The method, the scores, the intervals and the sample pages are public. Five public briefs run alongside the private ones as anchors, so a model that overfits the public set shows up as a gap.

Can a lab pay to be listed, rerun or removed?

No. We run every model ourselves at the provider’s default settings, through the lab’s own API or through OpenRouter, and record who served the pages. Labs don’t send outputs and can’t pick their best page. If a listed lab is a Rams customer, its row says so.

Can I use the numbers?

Yes. Every edition is published as a JSON file under /index/data, licensed CC BY 4.0: cite the Rams Index and link the board. Quote the interval with the score, and the rank range with the place.

How do I dispute a score?

Email rams@rams.ai with the page and what looks off. We render it again and judge it on the current method. If our render was wrong we fix or withdraw the score, and any change goes in the changelog with the reason.

Method 1.1 · updated 2026-10-16Questions or a disagreement: rams@rams.aiLab logos are trademarks of their owners, used to identify each model's vendor.