Introduction to Deep Learning

Babak Shahbaba

Why Deep Learning?

Face recognition and photo search
Speech recognition and translation
Large language models such as ChatGPT
Recommendations and autonomous vehicles

Today’s goal: build the core idea step by step, starting from regression.

Our Roadmap

Representation: how layers learn useful features
Training: forward pass, loss, and backpropagation
Evaluation: validation, testing, and overfitting
Architecture: matching the network to the data

By the end, you should be able to explain what a neural network learns, how it learns, and why different problems call for different architectures.

Generalized Linear Models (GLM)

Regression models attempt to learn a map from predictors \(x\) to a response \(y\):

\[(x_1, \dots, x_p) \rightarrow y\]

For GLM, this map has a simple form:

\[\beta_0 + x^\top \beta = g(\mu), \quad \text{where } \mu \text{ is the mean of } y\]

From observed data, we estimate the parameters and make predictions:

\[g^{-1}(\hat\beta_0 + x^\top \hat\beta) = \hat\mu, \quad \text{or alternatively} \quad f(\hat\beta_0 + x^\top \hat\beta) = \hat\mu\]

For logistic regression:

\[f(\hat\beta_0 + x^\top \hat\beta) = \frac{\exp(\hat\beta_0 + x^\top \hat\beta)}{1 + \exp(\hat\beta_0 + x^\top \hat\beta)}\]

We choose parameters that minimize a loss (equivalently, maximize likelihood for many models).

Input → weighted sum + bias → activation → prediction

Generalized Linear Models (GLM)

The following is a schematic representation of a GLM.

The box takes an input \(x\), combines it using learned weights, and outputs a prediction \(\hat y\).

Generalized Linear Models (GLM)

Every piece has a job: the weights determine how inputs contribute, the bias shifts the result, and the activation function determines the output scale.

Adaptive and Expanded Basis

We start from the same box as before: one function turning x directly into y.

Adaptive and Expanded Basis

Now we add a step in between: x first produces an intermediate value z, and z is then used to produce y.

Adaptive and Expanded Basis

We don’t have to stop at one intermediate value — we can compute several at once, each capturing a different pattern in x.

Adaptive and Expanded Basis

Bundled together, those intermediate values form z: a vector of learned features built from x.

A Shallow Neural Network

This is a neural network: \(x\) goes in, a hidden layer computes learned features \(z\), and the output layer produces \(\hat y\).

Regression with One Predictor

Before classification, consider a simpler task: predict a continuous response \(y\) from one predictor \(x\).

library(nnet)

set.seed(42)
regression_data <- tibble(
  x = sort(runif(45, -2, 2)),
  y = sin(2.5 * x) + 0.25 * x + rnorm(45, sd = 0.28)
)

Regression with One Predictor

ggplot(regression_data, aes(x, y)) +
  geom_point(size = 2.5, color = "#183153") +
  labs(title = "A noisy nonlinear relationship") +
  theme_minimal(base_size = 16)

The model must learn the shape of the relationship rather than a straight line.

What Will More Hidden Units Do?

We fit the same data three times, changing only the hidden-layer width:

\[3 \quad \longrightarrow \quad 5 \quad \longrightarrow \quad 10 \text{ hidden units}\]

Before seeing the result, predict:

Which model will have the lowest training error?
Which model can create the most bends in its fitted curve?
Does greater flexibility guarantee better predictions on new data?

More Hidden Units Increase Flexibility

set.seed(123)
hidden_units <- c(3, 5, 10)
prediction_grid <- tibble(x = seq(-2, 2, length.out = 400))

regression_fits <- map(hidden_units, \(units) {
  nnet(
    y ~ x, data = regression_data,
    size = units, linout = TRUE,
    decay = 0, maxit = 3000, trace = FALSE
  )
})

curve_data <- map2_dfr(regression_fits, hidden_units, \(model, units) {
  prediction_grid |>
    mutate(
      fitted = drop(predict(model, newdata = prediction_grid)),
      hidden_units = factor(
        paste(units, "hidden units"),
        levels = paste(hidden_units, "hidden units")
      )
    )
})

training_rmse <- map2_dfr(regression_fits, hidden_units, \(model, units) {
  tibble(
    hidden_units = units,
    rmse = sqrt(mean(
      (regression_data$y - drop(predict(model, regression_data)))^2
    ))
  )
})

More Hidden Units Increase Flexibility

Training RMSE: 3 units = 0.292; 5 units = 0.281; 10 units = 0.244.

Capacity Is Not the Same as Generalization

More hidden units give the model more ways to bend and adapt.
Training error generally decreases as capacity increases.
Some extra bends may follow random noise rather than the underlying signal.
Validation data, weight decay, and early stopping help control this flexibility.

The goal is not the most complex curve—it is the curve that predicts new observations best.

Example: Breast Cancer Wisconsin

We will predict whether a tumor is benign (B) or malignant (M).

This nnet model is deliberately small: it is a single-hidden-layer baseline, not a deep network. The same building blocks extend to many layers in frameworks such as Keras and PyTorch.

library(caret)

wdbc <- read_csv("wdbc.csv", show_col_types = FALSE) |>
  mutate(diagnosis = factor(diagnosis, levels = c("B", "M")))

count(wdbc, diagnosis)

# A tibble: 2 × 2
  diagnosis     n
  <fct>     <int>
1 B           357
2 M           212

Example: Breast Cancer Wisconsin

Use three separate roles:

Training: estimate weights
Validation: choose settings such as the decision threshold
Test: evaluate once at the end

set.seed(123)
train_index <- createDataPartition(wdbc$diagnosis, p = 0.60, list = FALSE)

train <- wdbc[train_index, ]
remaining <- wdbc[-train_index, ]

validation_index <- createDataPartition(
  remaining$diagnosis, p = 0.50, list = FALSE
)
validation <- remaining[validation_index, ]
test <- remaining[-validation_index, ]

preprocess <- preProcess(
  select(train, -diagnosis), method = c("center", "scale")
)
train_x <- predict(preprocess, select(train, -diagnosis)) |> as.matrix()
validation_x <- predict(
  preprocess, select(validation, -diagnosis)
) |> as.matrix()
test_x  <- predict(preprocess, select(test, -diagnosis)) |> as.matrix()

train_y <- as.integer(train$diagnosis == "M")
validation_y <- validation$diagnosis
test_y  <- test$diagnosis

Predict Before We Fit

Suppose we increase the hidden layer from 5 to 50 neurons:

What will happen to training error?
What might happen to validation error?
Would you expect a larger or smaller risk of overfitting?

Now suppose we lower the malignant-class threshold from 0.50 to 0.30:

Which should increase: sensitivity or specificity?
In cancer screening, which error is more costly?

Example: Breast Cancer Wisconsin

Fit one hidden layer with five neurons. Weight decay discourages unnecessarily large weights.

set.seed(123)
nn <- nnet(
  x = train_x, y = train_y,
  size = 5, entropy = TRUE,
  decay = 0.01, maxit = 1000, trace = FALSE
)

validation_prob <- drop(
  predict(nn, newdata = validation_x, type = "raw")
)

threshold_results <- map_dfr(seq(0.20, 0.80, by = 0.05), \(threshold) {
  validation_class <- factor(
    if_else(validation_prob >= threshold, "M", "B"),
    levels = levels(validation_y)
  )
  validation_cm <- confusionMatrix(
    validation_class, validation_y, positive = "M"
  )
  tibble(
    threshold,
    balanced_accuracy = mean(validation_cm$byClass[
      c("Sensitivity", "Specificity")
    ])
  )
})

selected_threshold <- threshold_results |>
  slice_max(balanced_accuracy, n = 1, with_ties = FALSE) |>
  pull(threshold)

prob_malignant <- drop(predict(nn, newdata = test_x, type = "raw"))
predicted_class <- factor(
  if_else(prob_malignant >= selected_threshold, "M", "B"),
  levels = levels(test_y)
)

cm <- confusionMatrix(data = predicted_class, reference = test_y,
                      positive = "M")
cm$table

          Reference
Prediction  B  M
         B 69  7
         M  2 35

Validation Chooses; Test Evaluates

The test set remains untouched until after this choice is fixed.

Example: Breast Cancer Wisconsin

Interpreting Performance

Accuracy: 92.0%
Sensitivity: 83.3%
Fraction of malignant tumors correctly detected
Specificity: 97.2%
Fraction of benign tumors correctly identified

The validation-selected threshold is 0.75. Lower thresholds usually detect more malignant tumors but create more false alarms.

Deep Neural Networks

A sufficiently wide network with one hidden layer can approximate any continuous function arbitrarily well—but it may be inefficient.
Depth lets a model compose simple features into more complex ones.
Modern deep learning was enabled by:
- larger datasets,
- GPUs and other accelerators,
- improved optimization and regularization methods.

Deep Neural Networks

Different activations give a network its nonlinearity. ReLU is the most common default for hidden layers; sigmoid is useful for binary output probabilities.

How Does the Network Learn?

Input (x)→ Hidden features (z)→ Prediction (y)→ Loss (L)

← gradients flow backward; parameters are updated ←

Forward pass: compute the prediction and loss.
Backpropagation: apply the chain rule to compute each gradient.
Optimizer: update the parameters:

\[\theta \leftarrow \theta - \eta\nabla_\theta L\]

The learning rate \(\eta\) controls how far the optimizer moves.

Loss Turns Mistakes into a Learning Signal

For binary classification, a common loss is cross-entropy:

\[L = -\frac{1}{n}\sum_i [y_i\log(\hat p_i) + (1-y_i)\log(1-\hat p_i)]\]

Confident correct predictions receive a small penalty.
Confident wrong predictions receive a large penalty.
Backpropagation identifies how every weight contributed to that penalty.

Learning means changing the weights to reduce future loss.

Generalization, Not Memorization

Training loss usually falls as model complexity increases.
Test performance may worsen when the model overfits.
Common safeguards:
- validation data and early stopping,
- weight decay (used in our example),
- dropout and data augmentation,
- enough representative training data.

Always evaluate the final model on data that were not used for training.

Autoencoder

Autoencoders compress an input into a latent representation and reconstruct it. They are useful for representation learning, denoising, and anomaly detection.

Convolutional Neural Networks (CNN)

CNNs share filters across locations, making them well suited to grid-like data such as images.

Transformers: “Attention Is All You Need”

Transformers use attention to combine information across positions. They power modern language models and are also used for images, audio, and biological sequences.

Source: https://arxiv.org/pdf/1706.03762

Transformers: “Attention Is All You Need”

Connection to ChatGPT:

GPT stands for Generative Pre-trained Transformer.
During pretraining, the model learns to predict the next token from the tokens that came before it.
At generation time, it repeatedly predicts a next-token distribution, selects a token, and feeds it back into the context.
Attention helps the model determine which earlier words or tokens are most relevant at each step.

Architecture Encodes Assumptions

Architecture	Useful assumption	Typical data or task
Fully connected	Every feature may interact with every other	Tabular data
CNN	Nearby patterns matter; the same pattern can appear anywhere	Images and spatial grids
Autoencoder	A compact representation can retain important structure	Compression, denoising, anomalies
Transformer	Relevant information may occur far away in the sequence	Text, audio, biological sequences

These built-in assumptions are called inductive biases. A good match can improve learning efficiency and generalization.

What Can Go Wrong?

Data hunger: large models may require many representative examples.
Computation: training can require substantial time, energy, and hardware.
Bias: a model can reproduce gaps and biases in its training data.
Interpretability: high accuracy does not guarantee an understandable reason.
Distribution shift: performance can deteriorate when future data differ from training data.

A strong test score is evidence—not a guarantee—of reliable deployment.

Takeaways

A neuron extends regression: weighted sum, bias, and activation.
Hidden layers learn useful intermediate representations.
Training alternates forward prediction with gradient-based updates.
Depth and architecture should match the structure of the problem.
Reliable evaluation requires clean splits and protection against overfitting.