Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Judge Quality

Which judges are best at spotting rising talent? The judge quality metric answers this by looking at a simple question: when a judge says "yes" to a dancer in prelims, does that dancer go on to level up?

How it works

For each judge, we look at every prelim panel they've ever judged:

  1. Find every dancer they gave a Yes to
  2. Check if those dancers later moved up to a higher skill level (e.g., from Novice to Intermediate)
  3. Score the judge based on how well their "yes" votes predicted future advancement

A judge who consistently says "yes" to dancers who later level up is identifying talent before the rest of the field catches on.

The score balances two things

A great talent-spotter needs to do both of these well:

  • Pick winners — When they say "yes," that dancer actually goes on to advance. A judge who says "yes" to everyone would pick all the future advancers, but also a lot of dancers who never advance.
  • Don't miss talent — Among all the dancers who did advance, how many did this judge recognize? A judge who only says "yes" to one dancer per night might be very accurate, but they'd miss most of the talent in the room.

The score rewards judges who do both — accurately identifying talent without being too conservative or too generous.

Experience matters

A judge who worked one panel and happened to vote for someone who advanced shouldn't outrank a veteran with hundreds of panels. Our ranking accounts for this:

  • Judges with many panels are ranked on their actual track record
  • Judges with few panels get a lower ranking because we're less certain about them
  • To be ranked highly, you need both good picks and enough data

What this is NOT

  • Not about agreeing with other judges. A judge who spots talent others miss would score higher, not lower.
  • Not about finals scoring. This only looks at prelim callbacks, where judges decide who advances through rounds.
  • Not a complete measure of judging ability. Judging involves artistry, musicality, technique, and many other factors that can't be captured in a single number.

Judge names

The same judge might appear in results as "John Smith" at one event and "John S." or "Johnny Smith" at another. We resolve these into a single identity so their full body of work is counted together.