Monday, July 14, 2014

Ratings & Difficulty: Appendix

In my last article, I tried to take an objective look at ratings. By far my most popular post ever, it was also, mostly well received. There were several good questions and comments that I will now address.

The Definition
I tried to make it clear that I was picking a definition for the words "hard" and "difficult." For the scope of my article, if twice as many people could do task 1 than task 2, then task 2 is referred to as twice as hard as task 1 (or twice as difficult.) Most people seemed happy enough with this approach; a few clearly hated it, but none offered alternatives.

The Goal
I was mainly trying to get an objective measure for how much harder (clearly using the above definition) one climbing grade was from the next. I was looking for the big-bucket (10%, 50%, twice, five times, ten times harder) rather than a discrete value. I knew the data available would only allow for a general range; something specific would require 10-100 times more data points.

The Data
The best source of data I could find was 8a.nu. This resulted in focusing only on the top end of the rating scale. Several people asked exactly how I got the data; here is my process:
  1. Go to www.8a.nu
  2. Click "Ranking"
  3. Click "All Time"
  4. Under the left column titled "All time Route Ranking", start with the first person on the list, click their name.
  5. When their profile comes up, click "Routes"
  6. Under "Ranking Routes" click "All Time"
  7. Copy the data from the "Route Scorecard Statistics" to Excel (or whatever)
  8. Go to step #4 and continue on with the next person on the list
"Missing" Data
Yes, I'm aware that Chris Sharma has climbed 9b+. I did not "exclude" him from the study. I studied only climbers that entered their routes into 8a and adding him would have tainted the data. This raised two subsequent issues/complaints, so let's look at each.
   First, many people asked how adding Chris would change the results (some even claimed that it would completely invalidate the entire article.) Below is the previous chart of the count of climbers that had climbed each grade with Chris added to the fifth column. (Remember, the way I looked at the data before, he counts as a climber that has climbed each of the grades, so each total increases by one.) Now we have 171 climbers that have climbed 8c+/14c turning into 2 that have climbed 9b+/15c. Following the same process as before, that means that 85.5 turns into 1 and the quad root of 85.5 is 3.04. The net result is that instead of saying these grades average 3.61 times harder than is previous, it would be 3.04 times harder — still the same major bucket.
Rating Climbers "Harder" 3.61 w/Chris 3.04
9b+/15c 1 2.0 1 2 1
9b / 15b 2 8.5 3.6 3 3.04
9a+ / 15a 17 4.9 13 18 9.24
9a / 14d 84 2.0 47 85 28.1
8c+ / 14c 170 170 171 85.4
The next issue is what happens if you add someone that has climbed 9b+/15c but not 9b/15b (or, for the sake of argument, no other grade.) That would mean there would be 2 9b+/15c climbers and 2 9b/15b climbers and thus, per our definition, no difference in difficulty, hence a logical flaw to some readers. This, clearly, is the problem of a lack of data, not a flawed assumption; any statistically large sample size will eliminate this "problem." (The math would be almost identical to the example above of adding Chris Sharma; it would be about a 3 times difficulty multiple.)

Standard Deviations
Several people commented that I should look at a normal distribution and increasing grades as steps from the mean. I'm hoping that someone with a better statistical background than myself will take up this torch, which is one of the reasons I supplied all my raw data. I looked at the data again in this light, and I'm convinced someone can generate something interesting here; but it's not me, at least, not right now. Ok, I'll take a guess... my guess is that the bell curve of climbing ability is in the ballpark of the one below, with 5.10c (redpointed outside) being the mean. The red line indicates how much harder each grade is from the previous one. I spent 15 minutes on this in Excel, so if I did something stupid, just relax.

Changes Over Time
Maybe the most thought provoking questions I got was this: "If we take data from 2002, when 15a was the top of the scale, but probably a lot less 14ds were in the log book, is there the same proportion of senders between 15a and 14d as there is today? Can grades get 'easier' over time?" I think this is a great question and I saved it for last since my response is simply my subjective opinion. I'll start by listing a few definitions of the word "technology":
  • the application of scientific knowledge for practical purposes
  • the practical application of knowledge especially in a particular area
  • a manner of accomplishing a task especially using technical processes, methods, or knowledge
It's important to remember that technology is not just your cell phone. Knowledge is technology. The routes we all climb are the most basic tool in making us better climbers, so they are a technology as well. So what does that mean when those routes don't exist? To me, it clearly means that you're going to have a harder time. This viewpoint will be unpopular as it makes it much harder to compare things over larger time scales.
   Think about it like this: is it easier to understand Einstein's theory of relativity today, or around 100 years ago? The theory (as published) hasn't changed, but only a handful of people understood it then whereas now every college physics student learns it. It just seems obvious that this is how things work: we find ways to make hard things easier and apply the time we "create" from this efficiency to work on something even harder.

Jackie Chan
Everyone's heard of Jackie Chan; but he did all his best work before ever being "discovered" in America. Below is the documentary that made me a fan — inspiring. I recommend: Police Story 1 & 3 (aka Super Cop,) Armor of God 1 & 2, Project A 1 & 2, Drunken Master 2, and Dragons Forever.