Decision Tree: Fill & Download for Free

I can think of many reasons.Ensembled decision trees generally perform better on heterogeneous data than neural networks.Ensembled decision trees are faster to train than neural networks.Ensembled decision trees don’t require expensive GPU servers.Ensembled decision trees are easier to deploy than neural networks.Ensembled decision trees require less hyperparameter tweaking than neural networks and the model architecture is pretty much fixed.Note that I said ensembled decision trees and not random forest, because I would usually prefer gradient boosted decision trees instead.These models are awesome for so many business use cases, because you can get good results quickly with relatively little effort.

The answer to "Should I ever use learning algorithm (a) over learning algorithm (b)" will pretty much always be yes. Different learning algorithms make different assumptions about the data and have different rates of convergence. The one which works best, i.e. minimizes some cost function of interest (cross validation for example) will be the one that makes assumptions that are consistent with the data and has sufficiently converged to its error rate.Put in the context of decision trees vs. logistic regression, what are the assumptions made?Decision trees assume that our decision boundaries are parallel to the axes, for example if we have two features (x1, x2) then it can only create rules such as x1>=4.5, x2>=6.5 etc. which we can visualize as lines parallel to the axis. We see this in practice in the diagram below.So decision trees chop up the feature space into rectangles (or in higher dimensions, hyper-rectangles). There can be many partitions made and so decision trees naturally scale up to creating more complex (say, higher VC) functions - which can be a problem with over-fitting.What assumptions does logistic regression make? Despite the probabilistic framework of logistic regression, all that logistic regression assumes is that there is one smooth linear decision boundary. It finds that linear decision boundary by making assumptions that the P(Y|X) of some form, like the inverse logit function applied to a weighted sum of our features. Then it finds the weights by a maximum likelihood approach.However people get too caught up on that... The decision boundary it creates is a linear* decision boundary that can be of any direction. So if you have data where the decision boundary is not parallel to the axes,then logistic regression picks it out pretty well, whereas a decision tree will have problems.So in conclusion,Both algorithms are really fast. There isn't much to distinguish them in terms of run-time.Logistic regression will work better if there's a single decision boundary, not necessarily parallel to the axis.Decision trees can be applied to situations where there's not just one underlying decision boundary, but many, and will work best if the class labels roughly lie in hyper-rectangular regions.Logistic regression is intrinsically simple, it has low variance and so is less prone to over-fitting. Decision trees can be scaled up to be very complex, are are more liable to over-fit. Pruning is applied to avoid this.Maybe you'll be left thinking, "I wish decision trees didn't have to create rules that are parallel to the axis." This motivates support vector machines.Footnotes:* linear in your covariates. If you include non-linear transformations or interactions then it will be non-linear in the space of those original covariates.

3 Problems with Decision TreesI illustrate by fitting a decision tree model in R to the "iris" dataset, which collects measurement data on 3 species of flowers. I focus on two of those measurements: sepal length and sepal width.library(rpart) library(rpart.plot) model1 <- rpart(Species ~ Sepal.Length + Sepal.Width, iris)  prp(model1, digits = 3) Now, I will perturb the data by adding 0.1 to each datapoint with probability 0.25, and subtracting 0.1 to each datapoint with probability 0.25.set.seed(1) tmp <- function() rbinom(nrow(iris), size = 1, prob = 0.5) perturb <- function() (tmp() - tmp()) / 10 iris$Sepal.Length.Perturbed <- iris$Sepal.Length + perturb() iris$Sepal.Width.Perturbed <- iris$Sepal.Width + perturb() model2 <- rpart(Species ~ Sepal.Length.Perturbed +  Sepal.Width.Perturbed, iris)  prp(model2, digits = 3) Key observation - Notice how just by perturbing the data a little bit, I made a different-looking decision tree?To get a better look at what's happening, I plot the decision tree boundaries and the actual data points on a scatter plot. I color each region by the plurality class.Some problem we see here when we apply our decision tree on continuous data:Instability - The decision tree changes when I perturb the dataset a bit. This is not desirable as we want our classification algorithm to be pretty robust to noise and be able to generalize well to future observed data. This can undercut confidence in the tree and hurt the ability to learn from it. One solution - Is to switch to a tree-ensemble method that combines many decision trees on slightly different versions of the dataset.Classification Plateaus - There's a very big difference between being on the left side of a boundary instead of a right side. We could see two different flowers with similar characteristics classified very differently. Some sort of rolling hill type of classification could work better than a plateau classification scheme. One solution - (like above), is to switch to a tree-ensemble method that combines many decision trees on slightly different versions of the dataset.Decision Boundaries are parallel to the axis - We could imagine diagonal decision boundaries that would perform better, e.g. separating the setosa flowers and the versicolor flowers.One very good method to reduce the instability is to rely on an ensemble of decision trees, by trying some sort of random forest or boosted decision tree algorithm. This also helps smooth out a classification plateau. An ensemble of slightly different trees will almost always outperform a single decision tree.If you prefer classification boundaries that aren't as rigid, you would also be interested in tree ensembles or something like K-Nearest-Neighbors.If you're looking for decision boundaries that are NOT parallel to the axis, you would want to try an SVM or Logistic Regression. See What are the advantages of logistic regression over decision trees? Are there any cases where it's better to use logistic regression instead of decision trees?For the other side of decision trees, see What are the advantages of using a decision tree for classification?