Syllabus
This syllabus is only indicative. Our aim is to cover topics for improving the performance of ML/AI programs.
Systems
- Models of ML systems: Shared memory, distributed memory
- Data organization: Numerics, Arrays;
- Machine organization: CPUs and GPUs;
- Compute abstractions: ISA, parallelism and its various forms, the out-of-order superscalar pipeline, threading, simultaneous multi-threading;
- Memory: hierarchy (caches), DMA, prefetching, coherence, technologies (SRAM, DRAM, HBM), virtual memory, paging, protection
- Storage: rotating media, solid-state media, network attached storage
- Network abstractions: topologies, collective communications
Programs
- DAG-based ML programming models
- High-performance Kernel Generation with DSLs and autotuning
- Parallel execution
- Distributed execution
- Pragmatics: Data management, scheduling, checkpointing, etc.
Efficiency
- Performance models
- Energy and power models
- Scalability: Amdahl's Law
Advanced Topics
- Non-von Neumann Accelerators: Dataflow (e.g. TPUs), deterministic (e.g., Groq)
- Recent advances in high-performance ML code generation