CS 464 • Final • Introduction to Machine Learning

Counting and Probability

Conditional Probability and Independence

Bayes' Rule

Discrete Random Variables

Continuous Random Variables

Expected Value and Variance

Bernoulli and Binomial Distributions

Continuous Uniform Distribution

Exponential Distribution

Normal Distribution

Laplace and Logistic Distributions

Motivation

Probabilistic Interpretation

Binary Cross Entropy / Log-loss

Optimization with Gradient Descent

Classification with Logistic Regression

Summary & Multi-Class Logistic Regression

Why Not Regression for Classification?

What & Why

Maximum Margin Classification

Maximizing the Margin

Lagrangian Formulation of the Hard-Margin SVM

From Primal to Dual: Solving the SVM Optimization

Why only a few points matter (KKT & sparsity)

From 𝛼 to parameters

Prediction uses only support vectors

Soft Margin SVM

Soft Margin Dual

Perceptron

Training a Perceptron

Limitation: XOR

MLP Architecture & Representation View

Backpropagation

Regression

Discrimination

MLP with Hard-Threshold Units

Should we initialize all MLP weights to zero?

Introduction to Deep Learning & Activation Functions

Training Deep Networks

Regularization Techniques

Tuning Network Structure

Learning Time

Time-Delay Neural Networks (TDNN)

RNN / LSTM / GRU

Generative Adversarial Networks (GANs)

What is a Decision Tree?

Splitting in Classification Trees

Pruning Trees

From Trees to Rules

Multivariate/Oblique Trees

Choosing Between Two Splits: Gini vs. Misclassification

Why Combine Multiple Learners?

Voting & Linear Combination

Bayesian Perspective & Effect of Dependence

Fixed Combination Rules & ECOC

Bagging & AdaBoost

Mixture of Experts and Stacking

Fine-Tuning an Ensemble

Cascading

Combining Multiple Sources/Views

Introduction and Mixture Densities

K-Means Clustering

K-Means Iteration Calculation

Expectation-Maximization (EM)

Mixture Models & Practical Use of Clusters

Spectral and Hierarchical Clustering

Pass Rates & Majors (Bayes; Law of Total Probability)

Weighted Least Squares (Closed-Form Solution, Matrix View & Interpretation)

MLE for α (positive support, exponential tail)

Linear Discriminant with Equal Variance

One Shared Network vs. Three Separate Networks

Naive Histogram Estimator vs. Parzen Windows (Kernel)

Kernel Smoother

Naive Density Estimator (Bandwidth effect & validity)

Comparing Two Splits (Gini vs. Misclassification)

Prepruning vs. Postpruning (Which and Why?)

From Binary to Multiclass: One-vs-All / One-vs-One with a Binary Classifier

Max-shift for SoftMax

Why Initialize Weights Near Zero?

Adaptive Learning Rates in Gradient Descent

When Do Direct Input Output Links Help in an MLP?

Mahalanobis vs. Euclidean: Why and When?

Discrete Attribute in Decision Trees

Regularized Least Squares

Gaussian Generative Model → Logistic Posterior

Naive Bayes Text Classification with Binary Features

Derivative of Softmax

Kernel Density Estimation