Tree Models

Learning objectives

This lecture examines whether an accurate prediction can be represented as a sequence of simple binary decisions.

By the end, you should be able to:

• read a decision tree as a piecewise prediction rule;
• explain conceptually how CART chooses a split;
• recognize overfitting and simplify a tree;
• evaluate predictive models; and
• explain how bagging and random forests work.

Decision trees as sequential rules

Consider a hypothetical classifier:

flowchart TD
  A{"Memory score < 4?"}
  B{"Age ≥ 75?"}
  C["Predict class 0"]
  D["Predict class 2"]
  E["Predict class 1"]
  A -- Yes --> B
  A -- No --> C
  B -- Yes --> D
  B -- No --> E

flowchart TD
  A{"Memory score < 4?"}
  B{"Age ≥ 75?"}
  C["Predict class 0"]
  D["Predict class 2"]
  E["Predict class 1"]
  A -- Yes --> B
  A -- No --> C
  B -- Yes --> D
  B -- No --> E

Each path from the root to a leaf is an if–then rule.

Trees divide the data into regions

A binary tree repeatedly divides the predictor space using rules such as age < 75 or memory score >= 4.

• Regression: the prediction is usually the mean response in a leaf.
• Classification: each leaf estimates class proportions and usually predicts the most common class.

Trees are therefore flexible piecewise-constant models.

Interpreting a classification leaf

Suppose a leaf contains 40 training observations:

The default prediction is class 2, but the useful output is often the full vector

\[ (\hat p_0,\hat p_1,\hat p_2)=(0.15,0.25,0.60). \]

Discussion: The largest estimated probability need not be the appropriate decision threshold when different errors have different consequences.

Choosing a split for regression

For regression, each leaf predicts the average outcome of the training observations in that leaf.

CART considers many possible variables and cutpoints. It prefers a split that creates child groups whose outcomes vary less than the outcomes in the parent group.

In other words, a useful split makes the observations within each child node more similar in their response values.

Choosing a split for classification

For classification, CART seeks child nodes that contain cleaner mixtures of classes.

• A pure node contains observations from only one class.
• A highly impure node contains a balanced mixture of classes.
• Gini impurity and entropy are two common measures of this mixture.

Choosing a split for classification

If \(p_c\) is the proportion of observations in class \(c\) within a node, then

\[ \text{Gini}=1-\sum_{c=1}^{C}p_c^2, \qquad \text{Entropy}=-\sum_{c=1}^{C}p_c\log(p_c). \]

Both measures approach zero as node purity increases. Therefore, CART chooses the split that produces the greatest improvement in purity.

Visualizing Gini impurity

For two classes, impurity is lowest when nearly every observation belongs to the same class and highest for a 50–50 mixture.

CART rewards splits that move child nodes toward the ends of this curve.

Worked example: comparing a split

Suppose a parent node has 60 observations: 30 in class A and 30 in class B.

A candidate split creates:

• left: 25 A and 5 B;
• right: 5 A and 25 B.

Both child nodes are much purer than the original 30–30 mixture, so this is a promising split. CART compares it with many other candidate variables and cutpoints.

Overfitting in large trees

If we keep splitting, leaves become increasingly specialized.

• Training error usually decreases.
• Test error may eventually increase.
• Small changes in the data can produce a different tree.

This is the bias–variance tradeoff:

• shallow tree: more bias, less variance;
• deep tree: less bias, more variance.

Pruning the tree

Pruning removes branches that add complexity without a reliable improvement in prediction.

In rpart, the complexity parameter cp controls how much improvement is required to keep a branch. Larger values produce smaller trees.

Cross-validation helps us choose a reasonable value.

Real data: cognitive diagnosis

We use a dataset with three diagnosis categories and several cognitive and demographic measurements.

alzheimer_raw <- read_csv(alzheimer_path, show_col_types = FALSE)

alzheimer <- alzheimer_raw %>%
  select(diagnosis, age, educ, female, memunits,
         trailb, bills, animals, naccmmse) %>%
  mutate(
    diagnosis = factor(diagnosis),
    female = factor(female)
  ) %>%
  drop_na()

dim(alzheimer)

[1] 2700    9

count(alzheimer, diagnosis) %>% mutate(proportion = n/sum(n))

# A tibble: 3 × 3
  diagnosis     n proportion
  <fct>     <int>      <dbl>
1 0          1534      0.568
2 1           613      0.227
3 2           553      0.205

The unequal class proportions mean accuracy alone may be misleading.

Training and test sets

We reserve 20% of the data for a final test.

set.seed(123)
train_id <- createDataPartition(
  alzheimer$diagnosis, p = 0.80, list = FALSE
)

train_data <- alzheimer[train_id, ]
test_data  <- alzheimer[-train_id, ]

prop.table(table(train_data$diagnosis))

        0         1         2 
0.5679926 0.2271045 0.2049029 

prop.table(table(test_data$diagnosis))

        0         1         2 
0.5687732 0.2267658 0.2044610 

createDataPartition() stratifies the split so class proportions remain similar.

Growing the initial tree

large_tree <- rpart(
  diagnosis ~ ., data = train_data, method = "class",
  control = rpart.control(cp = 0, minsplit = 10, xval = 10)
)

large_tree$cptable %>%
  as.data.frame() %>%
  as_tibble(rownames = "subtree") %>%
  slice_head(n = 15)

# A tibble: 15 × 6
   subtree      CP nsplit `rel error` xerror   xstd
   <chr>     <dbl>  <dbl>       <dbl>  <dbl>  <dbl>
 1 1       0.337        0       1      1     0.0247
 2 2       0.0610       1       0.663  0.663 0.0225
 3 3       0.0460       2       0.602  0.593 0.0217
 4 4       0.0178       3       0.556  0.561 0.0213
 5 5       0.0112       6       0.502  0.530 0.0209
 6 6       0.00910      8       0.480  0.528 0.0209
 7 7       0.00749     10       0.461  0.516 0.0207
 8 8       0.00642     11       0.454  0.520 0.0208
 9 9       0.00535     12       0.448  0.512 0.0207
10 10      0.00321     16       0.426  0.524 0.0208
11 11      0.00268     21       0.409  0.515 0.0207
12 12      0.00214     25       0.394  0.516 0.0207
13 13      0.00187     44       0.353  0.525 0.0208
14 14      0.00178     53       0.331  0.524 0.0208
15 15      0.00161     57       0.323  0.529 0.0209

The table records tree size, training error, cross-validated error (xerror), and its uncertainty (xstd). Focus on how validation error changes as the tree becomes more complex.

Minimum-error tree

Choose the cp with minimum cross-validated error:

best_row <- which.min(large_tree$cptable[, "xerror"])
best_cp <- large_tree$cptable[best_row, "CP"]
best_cp

[1] 0.005353319

The tree with the smallest cross-validated error is a natural choice. However, the estimated errors vary because cross-validation uses different subsets of the data.

Cross-validated error and tree complexity

Cross-validated error and tree complexity

The vertical axis is relative cross-validated error. The dashed horizontal line marks the one-standard-error threshold; the rule selects the smallest tree whose error falls below it.

Discussion: A slightly less accurate but substantially smaller tree may offer greater stability and interpretability.

The pruned tree

Each leaf reports the predicted class, class probabilities, and percentage of training observations reaching that leaf.

Class-probability predictions

tree_prob <- predict(pruned_tree, newdata = test_data, type = "prob")
tree_class <- predict(pruned_tree, newdata = test_data, type = "class")

head(tree_prob)

          0         1          2
1 0.8855932 0.1033898 0.01101695
2 0.3471698 0.5622642 0.09056604
3 0.8855932 0.1033898 0.01101695
4 0.2968750 0.4765625 0.22656250
5 0.3471698 0.5622642 0.09056604
6 0.8855932 0.1033898 0.01101695

The probability vector tells us much more than the final class.

• \((0.98,0.01,0.01)\): confident prediction
• \((0.35,0.34,0.31)\): uncertain prediction

Uncertainty matters, especially when errors have unequal consequences.

Confusion matrix

Rows and columns answer different questions.

tree_cm <- confusionMatrix(tree_class, test_data$diagnosis)
tree_cm$table

          Reference
Prediction   0   1   2
         0 267  39   6
         1  39  72  31
         2   0  11  73

Performance beyond overall accuracy

tree_cm$overall[c("Accuracy", "Kappa")]

 Accuracy     Kappa 
0.7657993 0.5950731 

tree_cm$byClass[, c("Sensitivity", "Specificity", "Balanced Accuracy")]

         Sensitivity Specificity Balanced Accuracy
Class: 0   0.8725490   0.8060345         0.8392918
Class: 1   0.5901639   0.8317308         0.7109474
Class: 2   0.6636364   0.9742991         0.8189677

• Sensitivity for class \(c\): among actual class-\(c\) cases, how many did we detect?
• Specificity: among cases not in class \(c\), how many did we reject?
• Balanced accuracy: average of sensitivity and specificity.
• Kappa: agreement beyond what chance agreement from class frequencies would suggest.

Metrics must match the scientific costs of errors.

ROC curves and decision thresholds

A classifier predicts probabilities before those probabilities are converted into classes. Changing the decision threshold changes the balance between sensitivity and specificity.

An ROC curve plots, across many possible thresholds:

• true-positive rate: sensitivity;
• false-positive rate: \(1-\text{specificity}\).

ROC curves and decision thresholds

A curve closer to the upper-left corner represents better discrimination. The diagonal line represents performance similar to random guessing.

The area under the curve (AUC) summarizes discrimination:

• AUC \(=0.5\): little ability to distinguish the two groups;
• AUC \(=1\): perfect separation.

ROC curves for the pruned tree

Because diagnosis has three classes, we construct a one-versus-rest curve for each class: one diagnosis category is treated as positive and the other two are combined as negative.

ROC curves for the pruned tree

ROC curves evaluate ranking across thresholds.

They do not determine which threshold is most appropriate.

That choice depends on the relative consequences of false negatives and false positives.

Advantages of tree models

• Human-readable decision rules.
• Naturally model nonlinearities and interactions.
• No need to standardize numeric variables.
• Work with numeric and categorical predictors.
• Automatic variable selection.

But interpretability decreases quickly as the tree grows.

Limitations of a single tree

• High variance: small data changes can alter early splits.
• Greedy search can miss a better future sequence of splits.
• Axis-aligned, piecewise-constant predictions can be crude.
• Standard trees do not extrapolate smoothly in regression.
• Split selection can favor predictors with many possible cutpoints.
• Predicted probabilities in small leaves can be unstable.

These weaknesses motivate ensemble models.

Why averaging helps

Different bootstrap samples produce somewhat different trees. Their individual errors will not occur in exactly the same places, so averaging their predictions reduces the influence of any one unstable tree.

Key idea: More trees help, but making trees less correlated can help even more.

Random forests and tree decorrelation

A random forest adds one modification to bagging:

At each split, consider only a random subset of predictors.

This prevents one dominant predictor from appearing at the top of nearly every tree.

• Bootstrap sampling randomizes observations.
• Feature subsampling randomizes candidate predictors.
• Averaging stabilizes the resulting predictions.

The tuning parameter mtry controls how many predictors are considered per split.

Out-of-bag evaluation

A bootstrap sample leaves out roughly one third of the distinct training observations.

These excluded observations are out of bag (OOB) for that tree.

For each training observation, aggregate predictions only from trees where it was OOB. This provides an internal estimate of generalization error without a separate validation set.

The untouched test set is still valuable for final evaluation.

Fit a random forest

set.seed(123)
rf_fit <- randomForest(
  diagnosis ~ ., data = train_data,
  ntree = 500, mtry = 2,
  importance = TRUE
)

rf_fit

Call:
 randomForest(formula = diagnosis ~ ., data = train_data, ntree = 500,      mtry = 2, importance = TRUE) 
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 2

        OOB estimate of  error rate: 19.57%
Confusion matrix:
     0   1   2 class.error
0 1144  75   9  0.06840391
1  164 240  87  0.51120163
2   14  74 355  0.19864560

The printed confusion matrix is based on OOB predictions for the training observations—not the final test set.

Test-set model comparison

rf_class <- predict(rf_fit, newdata = test_data, type = "response")
rf_cm <- confusionMatrix(rf_class, test_data$diagnosis)

tibble(
  model = c("Pruned tree", "Random forest"),
  accuracy = c(unname(tree_cm$overall["Accuracy"]),
               unname(rf_cm$overall["Accuracy"])),
  kappa = c(unname(tree_cm$overall["Kappa"]),
            unname(rf_cm$overall["Kappa"]))
)

# A tibble: 2 × 3
  model         accuracy kappa
  <chr>            <dbl> <dbl>
1 Pruned tree      0.766 0.595
2 Random forest    0.779 0.613

Use class-specific metrics too: an average improvement may hide a worse result for a clinically important class.

Random-forest confusion matrix

Compare: Which errors did the forest reduce? Which remain difficult?

Interpretation and limitations of variable importance

Permutation importance asks:

How much does prediction degrade when one predictor is shuffled?

rf_importance <- importance(rf_fit, type = 1, scale = TRUE) %>%
  as.data.frame() %>%
  rownames_to_column("variable") %>%
  as_tibble() %>%
  rename(importance = MeanDecreaseAccuracy) %>%
  mutate(
    variable = recode(
      variable,
      age = "Age",
      educ = "Years of education",
      female = "Sex",
      memunits = "Memory units",
      trailb = "Trail Making Test B",
      bills = "Ability to manage bills",
      animals = "Animal naming score",
      naccmmse = "MMSE score"
    ),
    variable = fct_reorder(variable, importance)
  )

Interpretation and limitations of variable importance

ggplot(rf_importance, aes(importance, variable)) +
  geom_col(fill = "#2C7FB8", width = 0.7) +
  geom_text(aes(label = round(importance, 1)),
            hjust = -0.15, size = 4) +
  expand_limits(x = max(rf_importance$importance) * 1.12) +
  labs(
    title = "Random-forest permutation importance",
    subtitle = "Larger values indicate a greater loss of accuracy after shuffling",
    x = "Mean decrease in prediction accuracy",
    y = NULL
  )

Interpretation and limitations of variable importance

Variables near the top of the chart contribute most strongly to the forest’s predictive performance. A value near zero indicates that shuffling the variable has little effect on accuracy.

Important caveats:

• importance is not causality;
• correlated predictors can share or mask importance;
• rankings can vary across resamples;
• importance does not show the direction of an effect.

Interpretability vs. predictive stability

The best model depends on whether the goal emphasizes explanation, prediction, or both.

Summary

• A tree is a piecewise model built from recursive binary questions.
• CART greedily chooses splits that reduce squared error or class impurity.
• Large trees fit training data well but can have high variance.
• Cross-validation and cost-complexity pruning control tree size.
• Bagging reduces variance by averaging trees fitted to bootstrap samples.
• Random forests further reduce correlation through random feature selection.
• Honest evaluation requires untouched test data and metrics matched to the consequences of errors.

A tree explains one decision path; a forest makes that decision more stable.