Referee Efficiency Ratings
- 21 Comment
Last Saturday I attended the MIT Sloan Sports Analytics Conference. During the Basketball Analytics panel, this quote from Mark Cuban got me thinking:
There’s not 10 players on the court, there’s 13. And three of them determine about 80 percent of what happens out there.
Along with this excerpt, he mentioned something along the lines of: “so if you’re not looking at the refs then you’re missing out on a lot.”
The point I get is, if we don’t understand refs, then we don’t understand the game.
With this, he makes a very good point. I think Mark gets a bad rap for wanting to talk about refs, when in reality we should be talking about refs. With all of the on/off-court work we’re doing, refs seem to be an obvious component to add. They absolutely are on the court. And they’re on the court a lot.
So why, then, have we not been looking at refs? I can come up with a few quick reasons:
- Refs don’t get credit for points in the box score, so ESPN doesn’t highlight their contributions on SportsCenter.
- Your favorite team can’t sign and/or trade refs.
- We don’t give MVP awards to refs.
- Refs don’t dunk or hit the game winning shot.
There are certainly many more, I bet, but that really isn’t the point of this post.
The point of this post is to measure the relationship between referees and efficiency.
The Model
To measure this relationship, I fit the following model to data from the ’07-’08 season:
Efficiency = Intercept + HCA + O1 + … + On + D1 + … + Dn + R1 + … + Rn
Where Efficiency = Points Scored per 100 Possessions, the home court advantage HCA = 1 if the offensive team is at home, O1,…,On = 1 if the player is on offense, D1,…,Dn = 1 if the player is on defense, and R1,…,Rn = 1 if the referee is on the court. These variables are 0 otherwise.
The Results
The following spreadsheet lists the referee ratings:
http://spreadsheets.google.com/ccc?key=pLJimPjd7oqtkdHwDWJ0CZg
The following spreadsheet lists the offensive ratings:
http://spreadsheets.google.com/ccc?key=pLJimPjd7oquzr1K4x3rbQQ
The following spreadsheet lists the defensive ratings:
http://spreadsheets.google.com/ccc?key=pLJimPjd7oquv4qUgn1Rbbw
Interpreting the Results
To interpret these results, we want to think in terms of holding all of the other variables at some constant value. As an example, for the refs, we’d estimate that Derrick Stafford is associated with an increase in 7.8 points per 100 possessions he’s on the court.
That being said, what would we want a ref’s measured relationship be? In my mind we’d want their relationship to be 0, as this would indicate the referees all have the same relationship to efficiency. Based on these results, I think it’s safe to say this is not the case. I’m sure Mark Cuban would agree.
The astute reader will notice all of the refs have standard errors that indicate the coefficients are not statistically significant from 0. Thus someone might conclude refs (or coaches, for that matter) don’t have an impact. But given our knowledge of the game and these results, it is safe to say that these refs do not call everything the same way.
A final point to mention is that there are always 3 refs on the court at one time. Thus their combined impact is not likely to be 0. That being said, it might be worth treating refs as individual units to see what sort of relationship ref units have with efficiency.
What Things do Refs do Differently?
Just like player ratings, this model doesn’t tell us why this relationship exists. It doesn’t tell us what refs are doing differently from each other, it’s just telling us there is some difference.
In an attempt to quantify this sort of thing, I’ll be including refs in all of my future on/off-court work. Hopefully this will give further insight into the differences between refs.
21 Comments on this post
Trackbacks
-
kpelton said:
Ryan, my assumption was that the referee ratings would be centered around zero and the interesting question would be their relative positioning. How are we supposed to interpret the fact that all of them show up as positive factors on offense?
March 11th, 2009 at 3:52 pm -
Ryan said:
I didn’t bother to transform the ratings to show how they’re relative to each other. I think the fact that they were all measured to be positive is noteworthy. That being said, we can’t be all that confident that some of the lower results are not negative.
March 11th, 2009 at 4:03 pm -
kpelton said:
Sure, but to what zero are these numbers compared? It would seem the only possible interpretation is relative to no referees at all, but obviously that doesn’t make any sense, since we don’t have data with no referees.
March 11th, 2009 at 4:10 pm -
Ryan said:
The interpretation is with respect to the players and the home court advantage. If we were to fix these player and home court advantage variables at some constant value, we would expect the refs to be associated with an increase in the listed number of points per 100 possessions.
March 11th, 2009 at 4:14 pm -
kpelton said:
If you re-ran the regression without the referees included, would you get the same values for players (and home-court advantage)? Or is there a constant term that is getting eaten into by the referees when you include them?
March 11th, 2009 at 4:29 pm -
Ryan said:
Without refs, the intercept increases roughly 14 points. The home court is the same, while each player’s rating differs (after a quick scan) by less than 1 point.
March 11th, 2009 at 4:46 pm -
edkupfer said:
I am not going to comment on this post.
March 12th, 2009 at 1:32 pm -
Ryan said:
Ed, I think you meant to say: “This comment intentionally left blank.”
March 12th, 2009 at 1:36 pm -
Amnon said:
Something’s wrong with your numbers.
The values have much less scatter than predicted
by these statistical errors.
Probably the errors are ovreestimated (by a factor of
>2), or there are huge correlations between them (so they are meaningless).March 13th, 2009 at 12:43 am -
Ryan said:
What do you mean when you say “The values have much less scatter than predicted by these statistical errors.”?
March 13th, 2009 at 12:48 am -
Eli said:
What do these numbers look like if you go back to the seasons when crooked ref Donaghy was active? It would be interesting to see if his numbers stick out… or if any other refs stick out in a similar way.
March 13th, 2009 at 7:53 pm -
Robert Moskowitz said:
If 100 possessions means team possessions, then the table seems wrong to me because Kobe scores more than 16 points per game, and the lakers have somewhat fewer than 100 possessions per game, I believe. In the Lakers vs. Spurs Thursday night, the team took 83 shots and turned the ball over 11 times, for a total of 94 possessions. Kobe scored 23, and that’s very low for him.
If it means 100 possessions while the player is on the floor, the table is even more distorted to the low side, since Kobe didn’t play the full 48 minutes.
March 13th, 2009 at 10:03 pm -
Ryan said:
To calculate the expected points per 100 possessions you add 5 offensive players plus 5 defensive players plus 3 refs along with home court advantage and an intercept. This is saying that holding the other players and refs constant at some value, Kobe is associated with an increase of the listed points per 100 possessions when on offense.
March 13th, 2009 at 10:35 pm -
wtsaila said:
Hard to interpret: does it mean refs randomly award points to the offense? Since all ref efficiencies are positive it says more about the league officiating favoring the offense than individual refs, no? Another thing that disturb me, but have no idea if it is important, is that the standard error is the same for all refs.
March 14th, 2009 at 8:49 am -
Ryan said:
It is worth examining. I felt like this shows there is a difference among refs. It can be interpreted in other ways. One interpretation is that this shows nothing.
By their, nature refs are going to eventually add points thanks to the foul situation just by being out there that can’t be attributed to the players. Or can it be attributed to players?
Ultimately refs make calls, and this is the area that should really be examined. Shooting fouls, turnovers, etc.
March 14th, 2009 at 10:22 am -
Mike Nicholas said:
I am unsure how old this string is, but I have a couple of thoughts regarding referees.
1. Referees/umpires/officials, etc. should be invisible. Meaning, the only time they are even noticed is when something goes wrong. Other than that, their roles should be much more like sidelines. There for all to see, but nothing more than part of the landscape. That’s the theory, anyhow.
2. Referees should be anonymous. Why? Because referees, unlike players, can only have a negative consequence. If they have a positive consequence (which is 99.9% of the time), that means everything’s gone right and there have been no controversial calls. And nobody notices this because, of course, there are no controversies that stand out. Of course, that is the perfect scenario. However, being human, they are not perfect, and can never be exepcted to BE perfect. So, placing names alingside their job titles does not do enything to improve any performance. Rather, it can serve as an intimidation factor for a rabid home team, possibly changing the outcome of a game, as the refs fear for their safety.
3. Naming referees will eventually lead to less quality of referees. Just think about it…if you are a referee and your personal information can be placed on any website, for any nut fanatic to track down and use for harrassment or intimidation, then your performance will be effected. Just think back to the 80’s when that baseball umpire called a runner safe when he was clearly out (I cannot recall the spcifics, other than the St. Louis Cardinals and Jack Clark were involved). This guy was tracked down (and this is before the Internet, folks) and his family harrassed for years. He had to move twice. I submit that this possibility leads to a number of high quality refs that refuse to go to the NBA, due to family caoncerns.
Though I like the idea of really bearing down on the referees and making them more accountable for their performance, I also see the other side, and some serious downsides. We want the best referees, with the fewest second thoughts, clearly focused on the call at the moment. We don’t want refs that are worried about making the wrong call due to harrassment and family issues. We don’t want them thinking. Like good players that have practiced well and long, we want referees reacting to the play solely on their experience and circumstance. Not thinking abot ancillary issues or possibilities.
Anhonw, there’s my $.02, for what it’s worth.
March 14th, 2009 at 10:54 am -
Ryan said:
You bring up good points Mike. As you suggest, there will always be some “bad” calls made (since refs aren’t human).
I think my idea with all of this is how are refs inherently different? Not that they make bad calls, but do some refs call things a little bit differently?
Just from watching you can get the sense that some refs allow more physical play than others. Is this really tractable, or something our mind tricks us into believing?
March 14th, 2009 at 11:17 am -
pcrackenhead said:
I wonder what it would be like to run these numbers with a comparison of how long each ref has been in the league.
Would a more experienced ref add less points, or more points, to a game?
March 18th, 2009 at 4:12 pm -
Gabe said:
You can’t just dismiss the notion of statistical significance like that. The fact that you can’t say for sure the nature of the association (positive or negative) is, in fact, a HUGE deal.
March 28th, 2009 at 10:22 pm -
Ryan said:
Gabe, thanks for the feedback. I certainly have a lot to learn, but in this case I was going with something I read by a smart guy. I probably interpreted it incorrectly and/or abusing the use of it.
As one smart person e-mailed, this really isn’t the right way to look at refs anyway.
March 30th, 2009 at 3:09 pm
[…] explaination for what happened. Either that, or it should start taking Mark Cuban’s idea of Referee Efficiency Ratings […]