Re: [R] GLM output for deviance and loglikelihood

From: Jeffrey Pollock <jpollock_at_williamhill.co.uk>
Date: Thu, 21 Apr 2011 10:30:33 +0100


First of all thank you both for the replies, my understanding is a lot clearer after thinking about the example you showed.

One further question though, looking at the code for 'polr' in MASS suggests that ordinal (and I would guess nominal too) data with levels >2 (ie not binary) also has a saturated model log-likelihood of 0;

> res <- optim(s0, fmin, gmin, method = "BFGS", hessian = Hess, ...)
> .
> .
> .
> deviance <- 2 * res$value

So am I right in saying that Binary data isnt the only case where this is true? It would make sense to me that for a multinomial model you could have a unique factor for each data point and thus be able to create a likelihood of 1.

I have an example of this similar to the poisson one;

> library(nnet)
> y <- sample(1:3,replace=TRUE,size=10)
> factor <- as.factor(1:10)
> mod <- multinom(y~factor)
# weights: 33 (20 variable)
initial value 10.986123
iter 10 value 0.073035
final value 0.000086
converged
> mod

Call:
multinom(formula = y ~ factor)

Coefficients:
  (Intercept) factor2 factor3 factor4 factor5 factor6 factor7 factor8 factor9 factor10 2 -14.33414 -21.50626 28.45961 28.45961 -9.18573 -9.18573 -9.18573 -9.18573 28.45961 -9.18573 3 -12.71240 -27.36982 -14.22512 -14.22512 23.75682 23.75682 23.75682 23.75682 -14.22512 23.75682

Residual Deviance: 0.0001713779
AIC: 40.00017
> logLik(mod)

'log Lik.' -8.568897e-05 (df=20)
> cbind(y,factor)

      y factor
 [1,] 1      1
 [2,] 1      2
 [3,] 2      3
 [4,] 2      4
 [5,] 3      5
 [6,] 3      6
 [7,] 3      7
 [8,] 3      8
 [9,] 2      9
[10,] 3     10

My understanding of this is that; if I observe a factor of '1' then the model will say with probability 1 that the outcome will be 1 and so on for the other rows in the dataset, and this shows in the estimated coefficients. I think the reason the log-likelihood doesn’t return exactly 0 is that the fitting algorithm used gets suitably close and then stops. Wouldn't make sense to continue the algorithm until the coefficients where either 'Inf' or '-Inf'.

Please let me know your thoughts on this.

Thanks again,

Jeff

-----Original Message-----
From: peter dalgaard [mailto:pdalgd_at_gmail.com] Sent: 21 April 2011 09:32
To: Juliet Hannah
Cc: Jeffrey Pollock; r-help_at_r-project.org Subject: Re: [R] GLM output for deviance and loglikelihood

On Apr 21, 2011, at 05:14 , Juliet Hannah wrote:

> As you mentioned, the deviance does not always reduce to:
>
> D = -2(loglikelihood(model))
>
> It does for ungrouped data, such as for binary logistic regression.

To be precise, it only happens when the log likelihood of the saturated model is 0, which for discrete models implies that the probability of the observed data under the saturated model is 1. Binary data is pretty much the _only_ case where this is true (because individual fitted probabilities become either zero or one).

--
Peter Dalgaard
Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501
Email: pd.mes_at_cbs.dk Priv: PDalgd_at_gmail.com

Received on Thu 21 Apr 2011 - 09:35:52 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Fri 22 Apr 2011 - 01:30:31 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive