Syllabus

This syllabus is only indicative. Our aim is to cover topics for improving the performance of ML/AI programs.

Systems

  1. Models of ML systems: Shared memory, distributed memory
  2. Data organization: Numerics, Arrays;
  3. Machine organization: CPUs and GPUs;
  4. Compute abstractions: ISA, parallelism and its various forms, the out-of-order superscalar pipeline, threading, simultaneous multi-threading;
  5. Memory: hierarchy (caches), DMA, prefetching, coherence, technologies (SRAM, DRAM, HBM), virtual memory, paging, protection
  6. Storage: rotating media, solid-state media, network attached storage
  7. Network abstractions: topologies, collective communications

Programs

  1. DAG-based ML programming models
  2. High-performance Kernel Generation with DSLs and autotuning
  3. Parallel execution
  4. Distributed execution
  5. Pragmatics: Data management, scheduling, checkpointing, etc.

Efficiency

  1. Performance models
  2. Energy and power models
  3. Scalability: Amdahl's Law

Advanced Topics

  1. Non-von Neumann Accelerators: Dataflow (e.g. TPUs), deterministic (e.g., Groq)
  2. Recent advances in high-performance ML code generation