Introduction to Parallel Programming

Unit 1: Introduction to Parallel Computing

• Trends in microprocessor architectures
• Memory system performance
• Dichotomy of parallel computing platforms
• Physical organization of parallel platforms
• Communication costs in parallel machines
• SIMD versus MIMD architectures
• Shared versus distributed memory
• PRAM shared-memory model
• Distributed-memory model

Unit 2: OpenMP programming for shared memory systems

• Thread Basics
• Controlling Thread and Synchronization Attributes
• Multi-thread and multi-tasking
• Context Switching
• Basic OpenMP thread functions
• Shared Memory Consistency Models and the Sequential Consistency Model
• Race Conditions
• Scoping variables
• work-sharing constructs
• critical sections
• atomic operations
• Introduction to Locks
• OpenMP tasks
• Introduction to tasks
• Task queues and task execution
• Accessing variables in tasks
• Completion of tasks and scoping variables in tasks
• Recursive task spawning and pitfalls

Unit 3: MPI programming for distributed memory systems

• Message Passing Interface (MPI) basic communication routines
• Introduction to Message Passing Interface (MPI) and basic calls
• MPI calls to send and receive data
• MPI call for broadcasting data
• MPI Non-blocking calls
• MPI Collectives
• MPI Collectives and MPI broadcast
• MPI Gathering and scattering collectives
• MPI reduction and Alltoall collectives
• MPI collectives design
• Types of interconnects
• Characterization of interconnects
• Linear arrays
• 2D mesh and torus
• Introduction to Cliques

Unit 4: Applications

• Matrix-matrix multiply
• Odd-Even sorting
• Distributed histogram
• Breadth First search
• Dijkstra’s algorithm