Difficulty - Revision history

Kevin Wang: /* Metrics */

2026-05-25T15:50:08Z

Metrics

Chris Holder: /* High School Level */

2025-04-07T19:14:37Z

High School Level

Kevin Wang: /* Metrics */

2023-11-23T00:05:29Z

Metrics

Kevin Wang: /* Metrics */

2023-08-28T01:38:00Z

Metrics

Kevin Wang: /* Gradations */

2023-08-28T01:33:57Z

Gradations

Kevin Wang: /* Gradations */

2023-08-28T01:32:53Z

Gradations

Kevin Wang: /* Metrics */

2023-08-28T01:27:16Z

Metrics

Kevin Wang: /* High School Level */

2023-08-27T20:31:06Z

High School Level

Kevin Wang: /* Relative */

2023-08-27T18:59:56Z

Relative

Kevin Wang: /* Relative */

2023-08-27T18:59:39Z

Relative

@@ Line 68: / Line 68: @@
 *These ranking systems are designed to estimate relative strength of teams and are typically judged by their ability to predict nationals performances or head-to-head match-ups.<ref>[https://www.hsquizbowl.org/forums/viewtopic.php?p=232529#p232529 Accuracy of HSQBRank] by [[AKKOLADE]] » Mon Jan 09, 2012 12:22 pm</ref> While they do as well as one could hope for from the data available, none of these systems were intended to describe set difficulty and as such cannot be reasonably be expected to do so with the same degree of accuracy.
-There has not been an attempt to "adjustment"-style ranking of college teams in recent memory. Possible reasons include smaller field sizes, a lower number of tournaments to compare, and a general lack of interest.
+There has not been an attempt at "adjustment"-style ranking of college teams in recent memory. Possible reasons include smaller field sizes, a lower number of tournaments to compare, and a general lack of interest.
 == Middle School Level ==

@@ Line 49: / Line 49: @@
 ==High School Level==
-At the high school level, [[HSAPQ]] tournament sets and [[IS|NAQT IS]] sets are considered the standard for regular difficulty. Most other sets are described in terms of how much easier or harder than these sets a tournament is expected to be.
+At the high school level, [[IS|NAQT IS]] sets are considered the standard for regular difficulty. Most other sets are described in terms of how much easier or harder than these sets a tournament is expected to be.
 ===Gradations===

@@ Line 66: / Line 66: @@
 *Both HSQBRank aPPB and Groger Rank bonus adjustments are derived from regression using the assumption that "adjustments between sets are purely additive"<ref>[[Steven Liu|Liu, Steven]]. “Groger Ranks 2019-20 Methodology Changes.” Groger Ranks, November 11, 2019. [https://grogerranks.files.wordpress.com/2019/11/gr_2019_methodology.pdf].</ref><ref>[[Fred Morlan|Morlan, Fred]]. “FAQ.” HSQBRank, September ?. [https://hsqbrank.com/faq/].</ref> - while empirical data has shown that this is approximately true in many cases, this is less true when conversion is above 20 PPB.
 *Adjustments are based on the comparison of [[IS]] sets and [[housewrite]]s. However, there is a huge difference in both field size and composition between the two kinds of set. For instance, in the 2022-23 season roughly the same number of schools on the "top 100" ranking published by Groger Ranks played the housewrites [[DART III]] and [[KICKOFF]] as [[IS-215]]. However, over three times as many total teams played IS-215, meaning that the ''fraction'' of top 100 schools at the housewrites was much higher (~70% vs. ~25%). Such a strong selection bias can have significant consequences on the observed performance of the field (and hence the adjustment given to the set), with some loose estimates suggesting it could totally mask the true difficulty of the set.<ref>[https://hsquizbowl.org/forums/viewtopic.php?p=395014#p395014 Re: Concern for HS Regs+ Sets this Season] by [[Santa Claus]] » Mon May 08, 2023 3:48 pm</ref>
-*These ranking systems are designed to estimate relative strength of teams and are typically judged by their ability to predict nationals performances or head-to-head match-ups.<ref>[https://www.hsquizbowl.org/forums/viewtopic.php?p=232529#p232529 Accuracy of HSQBRank] by [[AKKOLADE]] » Mon Jan 09, 2012 12:22 pm</ref> While they do as well one could hope for from the data available, none of these systems were intended to describe set difficulty and as such cannot be reasonably be expected to do so with the same degree of accuracy.
+*These ranking systems are designed to estimate relative strength of teams and are typically judged by their ability to predict nationals performances or head-to-head match-ups.<ref>[https://www.hsquizbowl.org/forums/viewtopic.php?p=232529#p232529 Accuracy of HSQBRank] by [[AKKOLADE]] » Mon Jan 09, 2012 12:22 pm</ref> While they do as well as one could hope for from the data available, none of these systems were intended to describe set difficulty and as such cannot be reasonably be expected to do so with the same degree of accuracy.
 There has not been an attempt to "adjustment"-style ranking of college teams in recent memory. Possible reasons include smaller field sizes, a lower number of tournaments to compare, and a general lack of interest.

@@ Line 61: / Line 61: @@
 ===Metrics===
-[[HSQBRank]] historically kept a set of "stat adjustments" which measured how teams performed on different packet sets: NAQT IS sets were normalized to zero, with positive numbers indicating a set played more difficult and negative numbers indicating that it played easier. [[Groger Ranks]], the spiritual successor to HSQBRank, has maintained similar numbers using its own system of assessing sets.
+[[HSQBRank]] historically kept a set of "stat adjustments" which measured how teams performed on different packet sets: NAQT IS sets were normalized to zero, with positive numbers indicating a set played more difficult and negative numbers indicating that it played easier. Its spiritual successor [[Groger Ranks]] has maintained similar numbers using its own system of assessing sets.
 These metrics are often taken to be a direct calculation of the relative difficulty of sets. This impulse should be tempered:
 *Both HSQBRank aPPB and Groger Rank bonus adjustments are derived from regression using the assumption that "adjustments between sets are purely additive"<ref>[[Steven Liu|Liu, Steven]]. “Groger Ranks 2019-20 Methodology Changes.” Groger Ranks, November 11, 2019. [https://grogerranks.files.wordpress.com/2019/11/gr_2019_methodology.pdf].</ref><ref>[[Fred Morlan|Morlan, Fred]]. “FAQ.” HSQBRank, September ?. [https://hsqbrank.com/faq/].</ref> - while empirical data has shown that this is approximately true in many cases, this is less true when conversion is above 20 PPB.
 *Adjustments are based on the comparison of [[IS]] sets and [[housewrite]]s. However, there is a huge difference in both field size and composition between the two kinds of set. For instance, in the 2022-23 season roughly the same number of schools on the "top 100" ranking published by Groger Ranks played the housewrites [[DART III]] and [[KICKOFF]] as [[IS-215]]. However, over three times as many total teams played IS-215, meaning that the ''fraction'' of top 100 schools at the housewrites was much higher (~70% vs. ~25%). Such a strong selection bias can have significant consequences on the observed performance of the field (and hence the adjustment given to the set), with some loose estimates suggesting it could totally mask the true difficulty of the set.<ref>[https://hsquizbowl.org/forums/viewtopic.php?p=395014#p395014 Re: Concern for HS Regs+ Sets this Season] by [[Santa Claus]] » Mon May 08, 2023 3:48 pm</ref>
 There has not been an attempt to "adjustment"-style ranking of college teams in recent memory. Possible reasons include smaller field sizes, a lower number of tournaments to compare, and a general lack of interest.

@@ Line 58: / Line 58: @@
 *The development of progressively better resources for studying like databases has hyper-accelerated the inclusion of information into the [[canon]]. A decade ago it would have taken a while for players to notice that questions on {{q|the {{bu|Mali}} Empire}} are largely the same year-to-year; in the modern day, that same observation can be made in one second by searching up the [[answerline]] on [[QBReader]]. Enfranchised players sometimes interpret this lack of novelty as a sign of lower difficulty (or lower quality<ref>[https://hsquizbowl.org/forums/viewtopic.php?p=384510#p384510 The majority of high school tossups on the Mali Empire are terrible.] by [[Vixor]] » Wed May 12, 2021 8:06 pm</ref>) rather than a natural consequence of writing a question for the expected level of knowledge of [[the field]].
-Difficulty modifiers like "+" and "-" are often used to specify these fine gradations (e.g. "regs-", "regs+", "nats-"). Comically-long strings of these modifiers are common in facetious examples of this system, like "nats-++-" ("nationals minus-plus-plus-minus").
+Difficulty modifiers like "+" and "-" are often used to specify these fine gradations (e.g. "regs-", "regs+", "nats-"). Comically-long strings of these modifiers are common in facetious examples of this system, like "nats-++-" ("nationals minus-plus-plus-minus") to describe a set infinitesimally easier than a "nats-++" set - the jury is out on whether this would be the same as "regs+--".
 ===Metrics===

@@ Line 52: / Line 52: @@
 ===Gradations===
-With few exceptions, high school quiz bowl tournaments span a range of difficulty roughly equivalent to the gap between [[NAQT Collegiate Novice]] and [[ACF Winter]] (i.e. one-third of the standard collegiate calendar, or between one-dot and two-dots on the collegiate calendar difficulty scale). Despite this, high school quiz bowl has an equal number of difficulty gradations (if not more). For instance, the difference between "nationals-minus" and "regs-plus" sets is often considered to have significant gameplay consequences on top high school teams; by contrast, collegiate "[[nats-minus]]" has more or less absorbed "Regs-plus".
+With few exceptions, high school quiz bowl tournaments span a range of difficulty roughly equivalent to the gap between [[NAQT Collegiate Novice]] and [[ACF Winter]] (i.e. one-third of the standard collegiate calendar, or between one-dot and two-dots on the collegiate calendar difficulty scale). Despite this, high school quiz bowl has an equal number of difficulty gradations (if not more). For instance, the difference between "nationals-minus" and "regs-plus" sets is often considered to have significant gameplay consequences on top high school teams; by contrast, collegiate "[[nats-minus]]" has more or less absorbed "Regs-plus" (though there are certainly those who would argue the same for these two levels).
 There are various reasons for this - here is an attempt at several of them:

@@ Line 47: / Line 47: @@
 [[Ophir Lifshitz]] has created a four-dot [https://collegequizbowlcalendar.com/difficulty-scale/ difficulty scale] to remove ambiguities in difficulty terminology.
-== High School Level ==
+==High School Level==
-At the high school level, [[HSAPQ]] tournament sets and [[IS|NAQT IS]] sets are considered the standard for regular difficulty. Most other sets are described in terms of how much easier or harder than these sets a tournament is expected to be. [[HSQBRank]] keeps a set of "stat adjustments" that measures the difficulty of different packet sets: NAQT IS sets are set to zero, while more positive numbers indicate more difficult sets and more negative numbers indicate easier sets.
+At the high school level, [[HSAPQ]] tournament sets and [[IS|NAQT IS]] sets are considered the standard for regular difficulty. Most other sets are described in terms of how much easier or harder than these sets a tournament is expected to be.
 == Middle School Level ==

@@ Line 30: / Line 30: @@
 * Even a perfect description of something's difficulty within the game will not be able to describe how hard it is in broader society. It is known (and frequently commented on) that the demographics of quiz bowlers are substantially different from the general population. Despite this, metrics like difficulty and [[importance]] often reference how well the average person (from the street or in a field) would know it.
-One can know the ''direction'' which stats have been skewed by these factors, but not the precise magnitude. These factors are often small enough (and the bins of "difficulties" are large enough) that most observers will broadly agree - for instance, even though some collegiate [[two-dot]] sets are harder than others, they are almost always closer to one another than they are to [[three-dot]] sets.
+One can typically figure out the ''direction'' which stats have been skewed by these factors, but not the precise magnitude. These factors are often small enough (and the bins of "difficulties" are large enough) that most observers will broadly agree - for instance, even though some collegiate [[two-dot]] sets are harder than others, they are almost always closer to one another than they are to [[three-dot]] sets.
 ==Regular difficulty==

@@ Line 28: / Line 28: @@
 * The difficulty of individual clues can be skewed considerably by appearing in other sets frequently, recently, or both. This is a major factor in why some questions play significantly easier after time has passed: inclusion of a piece of information into [[the canon]] makes it significantly easier for players who [[packet study|pay attention to it]].
 * Without some sort of comprehensive survey of all players, any single value will be incomplete in describing how difficult the community as a whole finds something.
-* Even a perfect description of something's difficulty within the game will not be able to describe how hard it is in broader society. It is known (and frequently commented on) that the demographics of quiz bowlers are substantially different from the general population. Despite this, metrics like difficulty and [[importance]] are often pinned to how well the average person (from the street or in a field) would know it.
+* Even a perfect description of something's difficulty within the game will not be able to describe how hard it is in broader society. It is known (and frequently commented on) that the demographics of quiz bowlers are substantially different from the general population. Despite this, metrics like difficulty and [[importance]] often reference how well the average person (from the street or in a field) would know it.
 One can know the ''direction'' which stats have been skewed by these factors, but not the precise magnitude. These factors are often small enough (and the bins of "difficulties" are large enough) that most observers will broadly agree - for instance, even though some collegiate [[two-dot]] sets are harder than others, they are almost always closer to one another than they are to [[three-dot]] sets.