Replies to This Discussion

Permalink Reply by Richard Boire on April 8, 2011 at 2:04pm

Sorry about that. pressed the post button too early. The more experienced approach as opposed to academic approach is as follows:

1)This is only a problem if the actual variable switches in sign in the multivariate routine from what you observe in correlation between the actual indep. variable and the dependant variable.

 

2)if this variable does switch in sign(very high multicollinearity), drop it and then see what other variable takes its place presuming no switching in sign with the new variable or variables and of course minimal compromise to your model's performance.

 

Hope this helps

 

Richard Boire

www.boirefillergroup.com

• ▶ Reply

Permalink Reply by Miles Garnsey on April 8, 2011 at 9:41pm

I not 100% sure about logistic regression, but for normal regression you should keep whichever variable has the strongest correlation, because it accounts for the most variance. If you have four all accounting for the same variance, it'll only make prediction harder because you won't know which variable is generating the predictions.

 

If they all account for a similar amount of variance, then I think you could really pick at random. Its not a hard and fast rule, but it is a good idea to make prediction more reliable and the model more stable.

 

But a better option might be to use some sort of dimensionality reduction tool like PCA etc to actually optimise the variance accounted for by each feature, especially if each is accounting for a decent amount of unique variance as well.

• ▶ Reply