Mathematical Foundations of Artificial Intelligence

Books

Required:

• NONE

Recommended I:

• [W] Larry Wasserman, All of Statistics, 2004.
• [DHS] Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Classification (2nd edition), 2001.
• [HTF] Trevor Hastie, Robert Tibshirani, Jerome Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd edition), 2009.

Recommended II:

• [B] Christopher M. Bishop, Pattern Recognition and Machine Learning.
• [KV] Michael J. Kearns and Umesh V. Vazirani, An Introduction to Computational Learning Theory, 1994.
• [M] Tom Mitchell, Machine Learning, 1997.
• [DGL] Luc Devroye, Laszlo Gyorfi, Gabor Lugosi, A Probabilistic Theory of Pattern Recognition, 1996.
• [A] Ethem Alpaydin, Introduction to Machine Learning, 2004.

Homework

1. (Probability theory) - due Jan. 28 - all exercises from handouts 1,2.
2. (Statistics I) - due Feb. 11 - all exercises from handout 3.
3. (Statistics II) - due Feb. 16 - all exercises from handout 4.
4. (Statistics III + EM + SVM's) - due Mar. 2 - all exercises from handouts 5,6,7,8.
5. (Graphical models + HMM's) - due Apr. 1 - all exercises from handouts 9,10.

Projects

1. Project 1 - due Mar. 15 11:59pm (send me .pdf + code in email) + printed version of the report (no code) next class (Mar. 16)

Lecture plan (tentative)

• Probability theory review (4 classes)
  
  READING: handouts 1,2, [W] Chapters 1,2,3,4,5.
  TOPICS: basics, inequalities (Markov, Chebyshev, Chernoff-Hoeffding), weak/strong law of large numbers, central limit theorem.
  
  
  • Jan. 14 (Th)
  • Jan. 19 (Tu)
  • Jan. 21 (Th)
  • Jan. 26 (Tu)
  
  
• Basic statistics (5 classes)
  
  READING: handout 3, [W] Chapters 6,7,8,9.
  TOPICS: confidence intervals, MLE, Bayesian inference, regression, classification, consistency, k-NN, EM algorithm, bootstrap.
  
  
  • Jan. 28 (Th)
  • class cancelled Feb. 2 (Tu)
  • class cancelled Feb. 4 (Th)
  • Feb. 9 (Tu)
  • Feb. 11 (Th)
  • Feb. 16 (Tu)
  • extra class Feb. 17 (Wed, 5pm-6:15pm, CSB 703)
  
  
• SVMs (2 classes)
  
  READING:
  TOPICS SVM's, duality theory, kernels.
  
  
  • Feb. 18 (Th)
  • Feb. 23 (Tu)
  
  
• Neural networks (1 class)
  
  READING:
  TOPICS: chain rule, straight-line programs, backpropagation.
  
  
  • extra class Feb. 24 (Wed, 5pm-6:15pm, CSB 703)
  
  
• Graphical Models (3 classes)
  
  READING:
  TOPICS: directed/undirected graphical models, elimination algorithm, treewidth, sum-product,max-sum algorithms.
  
  
  • Feb. 25 (Th)
  • Mar. 2 (Tu)
  • Mar. 4 (Th)
  
  
  • SPRING BREAK - Mar. 9 (Tu)
  • SPRING BREAK - Mar. 11 (Th)
    
    
  • MIDTERM I - Mar. 16 (Tu)
  
  
• Hidden Markov Models (2 classes)
  
  READING:
  TOPICS: HMMs, decoding (Viterbi's algorithm), evaluation, learning (EM algorithm), Kalman filters.
  
  
  • extra class Mar. 17 (Wed, 5pm-6:15pm, CSB 703)
  • Mar. 18 (Th)
  
  
• Reinforcement learning (2 classes)
  
  READING:
  TOPICS: Markov rewards process, Markov decision process, policy iteration, value iteration.
  
  
  • class cancelled Mar. 23 (Tu)
  • Mar. 25 (Th)
  • Mar. 30 (Tu)
  
  
• Component analysis (2 classes)
  
  READING:
  TOPICS: SVD, PCA, ICA.
  
  
  • extra class Mar. 31 (Wed, 5pm-6:15pm, CSB 703)
  • Apr. 1 (Th)
  
  
• Sampling (2 classes)
  
  READING:
  TOPICS: rejection sampling, MCMC, Gibbs sampler, Metropolis filter.
  
  
  • Apr. 6 (Tu)
  • extra class Apr. 7 (Wed, 5pm-6:15pm, CSB 703)
  
  
• Theory (3 classes)
  
  READING:
  TOPICS: PAC, VC dimension, uniform deviation inequalities, Boosting (adaboost).
  
  
  • Apr. 8 (Th)
  • Apr. 13 (Tu)
  • Apr. 15 (Th)
  • class cancelled Apr. 20 (Tu)
  • class cancelled Apr. 22 (Th)
  
  
  • MIDTERM II - Apr. 27 (Tu)

Grading

• Homework: 20%,
• Projects: 20%,
• EXAM 1: 30%,
• EXAM 2: 30%.

Homework/Project Rules

• Homework is collected on the due date, before class.
• No late homework will be accepted without instructor's permission (a permission must be requested at least 24 hours before the due date).
• In the final evaluation everybody's one lowest homework score and one lowest project score will be dropped.
• You may work with other people on the homework, but you must each write up your solutions separately (without any written aid, for example, looking at other people's source code (or written up solution) is forbidden). If you work with other people, indicate who you worked with on your solution.