Distributed data parallel training using Pytorch on AWS
In this post, I’ll describe how to use distributed data parallel techniques on multiple AWS GPU servers to speed up Machine Learning (ML) training. Along […]
Computer Architecture
Pipelining data processing and host-to-device data transfer
The purpose of this post is to show how to use multi-threading to parallelize data processing with data transfer from pageable to page-locked memory. I […]
Understanding Matrix Multiplication on a Weight-Stationary Systolic Architecture
If you follow the hardware for deep learning space, you may have heard of the term “systolic array”. A 2D systolic array forms the heart […]
Understanding Roofline Charts
The purpose of this post is to discuss my current understanding of roofline charts. Let me lay some background first. Before I got into machine […]