Sep 7 2009

A Basic Hierarchical Model of Efficiency

In my last post on retrodicting team efficiency, I set a general baseline that can be used help determine if a new model of team efficiency makes better predictions than a naive model. This is important, as we want to know if added model complexity is worth the hassle.

This post will present a basic hierarchical model of efficiency, and we’ll determine how much better this model is in terms of year to year predictions.

The Model

Like the classical adjusted plus/minus efficiency model, this basic hierarchical model for efficiency is constructed so that we make predictions about the mean number of points scored on a given possession between an offensive and defensive lineup. With this model, though, we consider all player’s offensive and defensive ratings to come from a normal distribution, where we consider two distributions: one for offensive ratings, and one for defensive ratings.

Mathematically, we can write this model as follows:

Y_{i} \sim {\tt N}(\beta_{0} + \beta_{1} + O_{1} + \cdots + O_{5} + D_{1} + \cdots + D_{5}, \sigma^{2}_{y}) \\ O_{i} \sim {\tt N}(0, \sigma^{2}_{o}) \\ D_{i} \sim {\tt N}(0, \sigma^{2}_{d})

Where \beta_{0} estimates the average number of points scored on a possession on the road, \beta_{1} estimates the home court advantage for when the offense is at home, and O_{i} and D_{i} estimate the offensive and defensive ratings for each player, respectively.

Also, because we have to use a Bayesian analysis, all of the parameters and hyperparameters of this model are given non-informative prior distributions.

The Player Ratings

The tables below list this model’s estimated top 10 offensive, defensive, and overall players from the 06-07, 07-08, and 08-09 seasons.

Rank06-07: NameORtg07-08: NameORtg08-09: NameORtg
1S. Nash5.0S. Nash6.5L. James5.7
2D. Nowitzki4.9C. Paul5.7C. Paul5.4
3C. Billups4.7K. Bryant5.4D. Wade5.3
4G. Arenas4.5D. Harris4.8M. Redd5.0
5L. Barbosa4.3D. Howard4.6L. Odom4.8
6C. Atkins4.3D. Nowitzki4.6S. Nash4.6
7A. Parker4.2L. James4.5K. Bryant4.2
8D. Lee4.1K. Martin4.5P. Stojakovic3.9
9M. Ginobili3.9P. Stojakovic4.4C. Billups3.8
10K. Bryant3.8M. Ginobili4.4A. Nocioni3.8
Rank06-07: NameDRtg07-08: NameDRtg08-09: NameDRtg
1C. Hayes-5.8K. Garnett-3.9K. Garnett-5.3
2B. Bowen-5.6C. Hayes-3.6A. Bogut-4.3
3D. Harris-5.2R. Wallace-3.5L. Odom-3.9
4N. Hilario-4.3J. Smith-3.4M. Camby-3.6
5T. Duncan-4.2J. O'Neal-3.3Y. Ming-3.4
6K. Garnett-4.0S. Pavlovic-3.1J. Przybilla-3.3
7R. Balkman-3.9J. Przybilla-3.0B. Wallace-3.0
8D. Mutombo-3.6M. Ginobili-2.9M. Jaric-3.0
9J. O'Neal-3.6A. Johnson-2.8A. Johnson-2.9
10K. Perkins-3.4R. Nesterovic-2.7B. Knight-2.9
Rank06-07: NameOvr Rtg07-08: NameOvr Rtg08-09: NameOvr Rtg
1D. Harris7.1M. Ginobili7.3L. Odom8.7
2T. Duncan7.1K. Bryant6.2L. James7.3
3K. Garnett7.1S. Nash6.2C. Paul7.2
4M. Ginobili6.6D. Nowitzki5.9D. Wade7.1
5B. Bowen6.1P. Pierce5.5K. Hinrich5.9
6A. Parker5.9J. Moon5.4K. Garnett5.4
7C. Hayes5.5P. Stojakovic5.1A. Kirilenko5.1
8S. Nash5.4C. Bosh5.1M. Bonner5.0
9D. Lee5.4T. Young5.0R. Lewis4.9
10D. Nowitzki5.2K. Garnett4.8Y. Ming4.8

As far as uncertainty in the ratings is concerned, we wouldn’t say that any one of these players is better than the other. Also, there are some cases in which we wouldn’t say that a given player is better than the average player.

Retrodiction Results

Obligatory top 10 lists aside, the real interest is in terms of predictions. Using the retrodiction methodology laid out in the post on retrodicting team efficiency, the ratings from the previous year’s model are used to predict the offensive, defensive, and net efficiency ratings of each team in the next season knowing game location and the 10 players on the court.

The following table lists the results:

Year to YearEfficiencyMeanStd DevMAEMAE DiffRMSERMSE Diff
06-07 to 07-08Offensive-0.023.732.91-1.793.66-1.22
06-07 to 07-08Defensive-0.583.552.64-0.973.54-1.34
06-07 to 07-08Net0.564.874.03-1.964.82-2.25
07-08 to 08-09Offensive-0.724.453.01-0.974.43-0.56
07-08 to 08-09Defensive-0.214.573.61-1.394.50-1.76
07-08 to 08-09Net-0.517.015.01-2.196.92-1.65

The difference in the classical model’s mean absolute error (MAE) and root mean squared error (RMSE) is listed under the MAE Diff and RMSE Diff columns. These are calculated by subtracting the classical model’s result from the hierarchical model’s result. In other words, negative numbers mean our predictions are doing better than the classical model.

Again, one area of interest is in the net efficiency. The graph below shows the predicted net efficiency versus the actual net efficiency:

Year to Year Net Efficiency Predictions

Unlike the classical model, this model does not predict teams to have very large or small net efficiency ratings, as most fall within +/- 10 points. This is probably a good thing, since most of the actual ratings fall within this range as well.

Summary

Based on the results above, I consider this basic hierarchical model to better predict efficiency over the classical model.

There is, however, still room for improvement, as we again have some weird results where we predict a net efficiency of -5, yet the actual net efficiency is 15.

Randomness will always give us imperfect results, but this result seems like one that is worth investigating in an effort to provide insight into how we can construct a better model of efficiency.

If you enjoyed this post, use RSS to get notified of new posts.

3 Comments on this post

Trackbacks


  1. Kevin Pelton said:

    Good stuff, Ryan. Two questions: Does the assumed normal distribution explain why these numbers are smaller than we see with classic adjusted plus-minus?

    Second, who’s that outlier at the top of the chart that this method actually predicted to be a below-.500 team?

    September 7th, 2009 at 10:02 pm
  2. Ryan said:

    The assumed normal distribution for player ratings keeps us from estimating ratings that are extreme (too big or too small) for players with small sample sizes. So this naturally keeps us from making extreme predictions for the future seasons.

    I believe the outlier you’re referring to is the one I mention in the summary. It is the 08-09 Bulls team, where predictions are only made for ~900 possessions since we ignore possessions from guys like D. Rose. I want to better understand why this collection of players did so well, and if we can explain it in any way.

    September 7th, 2009 at 10:14 pm
  3. Ryan said:

    I’ve constructed this spreadsheet that lists all offensive and defensive ratings from this model, with associated standard deviations and 95% credible intervals (which are constructed with the 2.5% and 97.5% quantiles).

    September 9th, 2009 at 9:55 pm