Parallelism in Deep Learning

Course Description

In this course, “Parallelism in Deep Learning,” you will learn about the need for parallelism in deep learning and how to use different methods of parallelism in deep learning. You will also learn about leveraging data parallelism and model parallelism workflows for your AI models on HPC infrastructures.

In this course, you will engage in hands-on activities, homework, and instructor consulting to make learning parallelism in deep learning enjoyable and rewarding. You will also be able to tackle real-world model training problems on HPC clusters. By the end of this course, you’ll have the skills and confidence to train your AI models at scale using multiple GPUs and nodes.

This course is one of 6 courses in the Advanced AI Techniques pilot Micro-Credential pathway offered by the Translational AI Center at Iowa State University.

Complete any 3 courses listed below to earn the Advanced AI Techniques badge:

• Scientific Machine Learning (SciML)
• Graph Neural Network
• Self-Supervised Learning
• Parallelism in Deep Learning
• 3D Vision – Nerfs & INRs
• Reinforcement Learning

Learn more about Micro-Credentials at Iowa State University!

Prerequisites

• Basic Python programming
• Basic understanding of deep learning
• Basic PyTorch programming

Intended Audience

The course is intended for a broad audience within the spectrum of the software and technology industry, including software engineers, data scientists, data engineers, data analysts, research scientists, and software developers. The course is designed to provide a basic understanding of high-performance computing for deep learning and how to use these models for a broad range of audiences.

Course Procedures

The course starts on March 3, 2025. All coursework must be completed by April 30, 2025, in order to earn the micro-credential badge. You will continue to have access to the course materials until January 1, 2026. The approximate time to complete this course is 16 hours. 

This course has an instructional period from March 3 to March 30, 2025. During this instructional period, course materials will be released weekly and live synchronous sessions will be held. You may complete the course materials at your own pace. Live Zoom meetings will be conducted for interactive coding sessions. A suitable time for these live sessions will be determined through a group poll. The recordings of those sessions will be available soon after each meeting.

You will receive the Parallelism in Deep Learning micro-credential badge upon successful completion of the course assessments.

Course Materials

Course materials are provided within the course. No additional purchase is required.

Contact Information

Contact isopd@iastate.edu for more information.

Course Developer

The Translational AI Center breaks down disciplinary silos to bring together core Iowa State artificial intelligence researchers and subject matter experts interested in applying new technologies to their work. For more information, visit Translational AI Center at Iowa State University

Instructor

Aditya Balu, Data Scientist

Aditya Balu is a data scientist in the Translational AI Center (TrAC) at Iowa State University. His research interests are in (i) Geometry-aware scientific machine learning (ii) Distributed and Decentralized Deep Learning (iii) Geometry-aware computational simulation tools.

Aditya also works on several topics in AI and its applications to diverse domains such as healthcare imaging, transportation, manufacturing, design, etc.

As part of TrAC, Aditya has organized several tutorials and workshops at reputed conferences such as CVPR, AAAI, and Supercomputing. He also has several publications in Neurips, ICML, CVPR and AAAI.

Learn more about this instructor