Replies to This Discussion

Permalink Reply by Stephen Cronin on June 21, 2010 at 12:36pm

http://bit.ly/bOzPuq -> will have to hunt for this for free. random forests of multinomial logits. 2008 good for multiclass classification and regression and overcoming curse of dim.

http://bit.ly/asrmLW -> BART. rigor of bayes, coolness of bagging. 2008 you can fire this up in R as well.

now I'm curious what the state of the art is in regression trees. Throw back some feedback if you find something else out there.

• ▶ Reply

Permalink Reply by Kyriacos Chrysostomou on July 15, 2010 at 10:37am

I am aware that the original documentation written by Brieman et al can be a little hard to get hold of (I mean, it is more than 20 years old), but there are a few papers and books out there that may be of interest to you:

1) Lee, M., Chrysostomou, K.A., Chen, S.Y., and Liu, X., (2008). Applications of Decision Trees for Data Modelling. Encyclopedia of Artificial Intelligence, 1, 437-442 – Gives an overview of the different types of decision trees including CART, and also the popular applications of such decision trees. [Only Abstract at http://www.igi-global.com/bookstore/Chapter.aspx?TitleId=10284. If you’re interested in the full article then please let me know and I might be able to send you the full version]
2) Kohavi, R. and Quinlan, R., (1999). Decision Tree Discovery. – Detailed and profound comparison of C4.5 and CART decision tree classifiers (I would highly recommend this as base reading for CART) [see the following link for full article: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.4.5353]
3) You may also want to read the book: Hand, D.J., Mannila, H., and Smyth, P., (2001). Principles of data mining, MIT Press, which gives a detailed description of all types of decision trees (including CART)

In your quest to learn about decision trees, in particular the CART classifier, please remember that all types of decision tree classifiers that you read about will more or less follow the same process: (1) splitting data using a so-called splitting criterion (2) forming the final decision tree, and (3) pruning the final tree to reduce its size and increase its classification abilities. So, if you understand how one decision tree classifier works, you will probably understand how all of them work.

However, the decision tree classifiers differ mainly in how they do Step 1 and Step 2. In terms of Step 1, decision tree classifiers may use different splitting criterion, for example the CART classifier uses a gini index to make the splits in the data (which only results in binary splits) as opposed to the information gain measure (which can result in two or more splits) like other tree classifiers use. Another major difference between decision tree classifiers is the type of data they can handle/process: CART can process both categorical and numerical data, while others can only handle categorical data.

Hope this helps you in your quest to find some useful and detailed information on CART.

Dr Kyriacos Chrysostomou

• ▶ Reply

Permalink Reply by DataMiner on July 27, 2010 at 6:21am

Dear Dr Kyriacos Chrysostomou,

I am working on decsion trees produced using CHAId. Can you please send the first article to me also, my email id is amithombre@gmail.com.....

Regards
Amit

• ▶ Reply

Permalink Reply by Lester Wollman on September 7, 2011 at 11:05am

There is a great book called "Elements of Statistical Learning" by Hastie, Tibshirani, and Friedman which describes in detail classification and regression trees and many other data mining methods.  You can buy the book or download it in pdf. The link is:

 

http://www-stat.stanford.edu/~tibs/ElemStatLearn/

 

Lester Wollman



• ▶ Reply

Permalink Reply by Gene Leynes on September 7, 2011 at 12:14pm

That book is amazing.  I was looking for something more specialized, but I'm not working on that problem anymore anyway.  Thanks for the input.

• ▶ Reply

Permalink Reply by Jozo Kovac on September 7, 2011 at 2:14pm

Isn't it funny that CART is one of the best algorithms 27 years after it was introduced? :-)

Logistic regression has similar story - very old and very useful for today's problems.

If you want to improve accuracy learn more about ensembles - boosting, bagging, adaboost, random forests. But don't expect miracles, they all improve CART's performance by relatively small margin. And all got some pros&cons.

http://www.salford-systems.com/doc/newhybridmethods.pdf - here's nice presentation about CART, only 13 years old, but really worth reading.

• ▶ Reply