High Performance Machine Learning (HPML)
Course Description
During the past decades, the field of High-Performance Computing (HPC) has been about
building supercomputers to solve some of the biggest challenges in science. HPC is where
cutting edge technology (GPUs, low latency interconnects, etc.) is applied to the solution of
scientific and data-driven problems.
One of the key ingredients to the current success of ML is the ability to perform computations on
very large amounts of training data. Today, the application of HPC techniques to ML algorithms
is a fundamental driver for the progress of Artificial Intelligence.
In this course, you will learn HPC techniques that are typically applied to supercomputing
software, and how they are applied to obtain the maximum performance out of ML algorithms.
You will also learn about techniques for building efficient ML systems. The course is based on
PyTorch, CUDA programming, MPI.
Objectives
At the end of the course, you will be able to:
●Use HPC techniques to find and solve performance bottlenecks
●Do performance measurements and profiling of ML software
●Evaluate the performance of different ML software stacks and hardware systems
●Develop high performance distributed ML algorithms
●Use fast math libraries, CUDA and C++ to accelerate High-Performance ML algorithms
●Model compression
Prerequisites
●Knowledge of computer architecture and operating system
●C/C++: intermediate programming skills
●Python: intermediate programming skills.
●Understanding of Machine Learning concepts and Neural Networks algorithms:
The course is focused on the system performance rather than the algorithms, and a basic
explanation of the algorithms will be part of the course. However, it is strongly recommended to
start the course with a good understanding of the following algorithms: logistic regression, feed
forward (basic) neural networks, convolutional neural networks, recurrent neural networks.
Course materials
The course does not follow a specific textbook, however some parts of the following
books can be used as a learning support. Pointers to specific literature/web links will be
provided in class.
Introduction to High Performance Computing for Scientists and Engineers
Authors: Georg Hager, Gerhard Wellein Editor: CRC Press
ISBN: 9781439811924
Introduction to High Performance Scientific Computing (ONLINE)
Authors: Victor Eijkhout with Edmond Chow, Robert van de Geijn
