Re: [R] Function for deleting variables with >=50% missing obs from a data frame

From: Rita Carreira <ritacarreira_at_hotmail.com>
Date: Mon, 18 Apr 2011 21:48:15 +0000

Thanks for the suggestion Daryl! I did have to include the exclamation point before mean, otherwise it selected the columns with the most missing observations. But it was really nice to see this flexibility in R. So my fix was dfQ<- dfQtemp[ , sapply(dfQtemp, function(x) !mean(is.na(x))>.6)] Thanks again!

Rita ===================================== "If you think education is expensive, try ignorance."--Derek Bok




> Date: Fri, 15 Apr 2011 15:08:29 -0700
> From: darylm_at_uw.edu
> To: ritacarreira_at_hotmail.com
> Subject: Re: [R] Function for deleting variables with >=50% missing obs from a data frame
>
> you could simply modify
>
> !all(is.na(x))
>
> to
>
> mean(is.na(x))> .6
>
> or some such, or invert the logic if I have it backwards.
>
> .6 was the fraction greater than which we omit the data.
>
>
>
>
> On 4/15/11 3:00 PM, Rita Carreira wrote:
> > Hello R users!
> > I have several data frames where some of the variables have many missing observations. For example, Q1 in one of my data frames has over 66% of its observations missing. I have tried imputation with mice but it does not work for all the data frames and I get the following message or a similar message to this:
> > iter imp variable
> > 1 1 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q19 Q36 Q47 Q52 Q79 Q80 Q94 Q97 Q104 Q108 Q122 Q131 Q134 P1 P2 P3 P4 P5 P6Error in solve.default(xtx + diag(pen)) :
> > system is computationally singular: reciprocal condition number = 1.83044e-16
> > In addition: Warning messages:
> > 1: In sqrt((sum(residuals^2))/(sum(ry) - ncol(x) - 1)) : NaNs produced
> > ...
> > 7: In sqrt((sum(residuals^2))/(sum(ry) - ncol(x) - 1)) : NaNs produced
> > Note: warnings 2 to 6 suppressed by me.
> > I would like to try a different approach where I delete the variables that have more than 50% missing observations from the data frame (well, the actual percentage might change). I have already deleted from the data frame the variables that were all missing and for this I used the following code, which was kindly suggested by one of you:
> > ## Data frame after removing any blank columns:dfQ<- dfQtemp[ , sapply(dfQtemp, function(x) !all(is.na(x)))]
> > Any ideas or suggestons for deleting variables with partially missing data?
> > Thanks and have a great weekend!
> > Rita ===================================== "If you think education is expensive, try ignorance."--Derek Bok
> >
> >
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help_at_r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
                                               

        [[alternative HTML version deleted]]



R-help_at_r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Received on Mon 18 Apr 2011 - 21:55:01 GMT

Archive maintained by Robert King, hosted by the discipline of statistics at the University of Newcastle, Australia.
Archive generated by hypermail 2.2.0, at Mon 18 Apr 2011 - 22:00:32 GMT.

Mailing list information is available at https://stat.ethz.ch/mailman/listinfo/r-help. Please read the posting guide before posting to the list.

list of date sections of archive