ENGR 421 • Midterm II • Introduction to Machine Learning

Counting and Probability

Conditional Probability and Independence

Bayes' Rule

Discrete Random Variables

Continuous Random Variables

Expected Value and Variance

Bernoulli and Binomial Distributions

Continuous Uniform Distribution

Exponential Distribution

Normal Distribution

Laplace and Logistic Distributions

What Nonparametric Means

Density Estimation

Nonparametric Classification

Condensed Nearest Neighbor

Outlier Detection

Nonparametric Regression

Additive Models & How to chose h/k

Histogram density estimator

Naive / uniform kernel density estimator

k-nearest neighbor (k-NN) classifier in 1D

What is a Decision Tree?

Splitting in Classification Trees

Pruning Trees

From Trees to Rules

Multivariate/Oblique Trees

Choosing Between Two Splits: Gini vs. Misclassification

What & Why

Maximum Margin Classification

Maximizing the Margin

Lagrangian Formulation of the Hard-Margin SVM

From Primal to Dual: Solving the SVM Optimization

Why only a few points matter (KKT & sparsity)

From 𝛼 to parameters

Prediction uses only support vectors

Soft Margin SVM

Soft Margin Dual

Margin, distance, and support vectors

Solving a tiny SVM dual problem (linear kernel)

Polynomial kernel and feature map

Dimensionality Reduction

Principal Component Analysis (PCA)

True/False on Feature Selection and PCA Fundamentals

Derivation of the PCA Objective via Lagrange Multipliers

Numerical Computation of the First Principal Component

Feature Embedding & Factor Analysis (FA)

Singular Value Decomposition and Matrix Factorization

Multidimensional Scaling

Linear Discriminant Analysis (LDA)

LDA Objective and Its Contrast with PCA

Canonical Correlation Analysis

Isomap, Locally Linear Embedding, Laplacian Eigenmaps

Forward vs Backward Selection Trade-offs

Linear Discriminant with Equal Variance

Naive Histogram Estimator vs. Parzen Windows (Kernel)

Kernel Smoother

Naive Density Estimator (Bandwidth effect & validity)

Comparing Two Splits (Gini vs. Misclassification)

Prepruning vs. Postpruning (Which and Why?)

Discrete Attribute in Decision Trees

Kernel Density Estimation

Naive Bayes Text Classification with Binary Features

Decision Trees: Gini Impurity Split Comparison

Decision Trees: Entropy & Information Gain Split Comparison

Kernel Engineering

1-NN LOOCV on Patient Dataset

k-NN Regression Prediction

Decision Boundary and Building a Network for Binary Classification

True/False on Scaling, k-NN, Intrinsic Error and Model Complexity