User’s Manual

• Introduction
• HPCToolkit Overview
  • Asynchronous Sampling and Call Path Profiling
  • Recovering Static Program Structure
  • Reducing Performance Measurements
  • Presenting Performance Measurements
• Quick Start
  • Guided Tour
    • Compiling an Application
    • Measuring Application Performance
      • Specifying CPU Sample Sources
      • Measuring GPU Computations
    • Recovering Program Structure
    • Analyzing Measurements & Attributing Them to Source Code
    • Presenting Performance Measurements for Interactive Analysis
    • Effective Performance Analysis Techniques
  • Additional Guidance
• Effective Strategies for Analyzing Program Performance
  • Monitoring High-Latency Penalty Events
  • Computing Derived Metrics
  • Pinpointing and Quantifying Inefficiencies
  • Pinpointing and Quantifying Scalability Bottlenecks
    • Scalability Analysis Using Expectations
      • Weak Scaling
      • Exploring Scaling Losses
• Monitoring Dynamically-linked Applications with hpcrun
  • Using hpcrun
    • If hpcrun causes your application to fail
      • hpcrun causes failures related to loading or using shared libraries
      • hpcrun causes your application to fail when gprof instrumentation is present
  • Hardware Counter Event Names
  • Sample Sources
    • Linux perf_events
      • Capabilities of HPCToolkit’s perf_events Interface
        • Frequency-based sampling.
        • Multiplexing.
        • Thread blocking.
      • Launching
      • Notes
    • PAPI
      • Proxy Sampling
    • REALTIME and CPUTIME
    • IO
    • MEMLEAK
  • Experimental Python Support
    • Known Limitations
  • Process Fraction
  • Starting and Stopping Sampling
  • Environment Variables for hpcrun
  • Cray System Specific Notes
• Monitoring Statically Linked Applications with hpclink
  • Linking with hpclink
    • Using hpclink when gprof instrumentation is present
  • Running a Statically Linked Binary
  • Troubleshooting
• Monitoring MPI Applications
  • Running and Analyzing MPI Programs
  • Building and Installing HPCToolkit
• Measurement and Analysis of GPU-accelerated Applications
  • GPU Performance Measurement Substrate
    • Profiling GPU Activities
    • Tracing GPU Activities
  • NVIDIA GPUs
    • Performance Measurement of CUDA Programs
    • PC Sampling on NVIDIA GPUs
    • Attributing Measurements to Source Code for NVIDIA GPUs
    • GPU Calling Context Tree Reconstruction
  • AMD GPUs
  • Intel GPUs
  • Performance Measurement of OpenCL Programs
• Measurement and Analysis of OpenMP Multithreading
  • Monitoring OpenMP on the Host
  • Monitoring OpenMP Offloading on GPUs
    • NVIDIA GPUs
    • AMD GPUs
    • Intel GPUs
• Analyzing Performance Data with hpcviewer
  • Launching
  • Profile View
  • Panes
    • Source Pane
    • Navigation Pane
      • Navigation Control
      • Context menus
    • Metric Pane
  • Understanding Metrics
    • How Metrics are Computed
    • Example
  • Derived Metrics
    • Formulae
    • Examples
    • Creating Derived Metrics
  • Metrics in Execution-context level
    • Plot Graphs
    • Thread View
  • Filtering Tree Nodes
  • Convenience Features
    • Editor Pane
    • Metric Pane
  • Trace view
    • Main View
    • Depth View
      • Specifying a new crosshair time.
      • Selecting a new time range.
    • Summary View
    • Call Stack View
    • Mini Map View
  • Menus
    • File
    • Filter
    • View
    • Help
  • Limitations
• Known Issues
  • Inaccurate attribution of performance information can occur on Sapphire Rapids processors with HPCToolkit
  • We observed bad generated code for Dyninst using gcc 11.2.0 on Aurora
  • When monitoring applications that use ROCm 6.0.0, using LD_AUDIT in hpcrun may cause it to fail to elide OpenMP runtime frames
  • When using Intel GPUs, hpcrun may report that substantial time is spent in a partial call path consisting of only an unknown procedure
  • hpcrun reports partial call paths for code executed by a constructor prior to entering main
  • hpcrun may fail to measure a program execution on a CPU with hardware performance counters
  • hpcrun may associate several profiles and traces with rank 0, thread 0
  • hpcrun sometimes enables writing of read-only data
  • A confusing label for GPU theoretical occupancy
  • Deadlock when using Darshan
• FAQ and Troubleshooting
  • Instrumenting Statically-linked Applications
    • Using hpclink with cmake
  • General Measurement Failures
    • Unable to find HPCTOOLKIT root directory
    • Profiling setuid programs
    • Problems loading dynamic libraries
    • Problems caused by gprof instrumentation
  • Measurement Failures using NVIDIA GPUs
    • Deadlock while monitoring a program that uses IBM Spectrum MPI and NVIDIA GPUs
    • Ensuring permission to use GPU performance counters
    • Avoiding the error cudaErrorUnknown
    • Avoiding the error CUPTI_ERROR_NOT_INITIALIZED
    • Avoiding the error CUPTI_ERROR_HARDWARE_BUSY
    • Avoiding the error CUPTI_ERROR_UNKNOWN
  • General Measurement Issues
    • How do I choose sampling periods?
    • Why do I see partial unwinds?
    • Measurement with HPCToolkit has high overhead! Why?
    • Some of my syscalls return EINTR
    • My application spends a lot of time in C library functions with names that include mcount
  • Problems Recovering Loops in NVIDIA GPU binaries
  • Graphical User Interface Issues
    • Fail to run hpcviewer: executable launcher was unable to locate its companion shared library
    • Launching hpcviewer is very slow on Windows
    • Mac only: hpcviewer runs on Java X instead of “Java 11”
    • When executing hpcviewer, it complains cannot create “Java Virtual Machine”
    • hpcviewer fails to launch due to java.lang.NoSuchMethodError exception.
    • hpcviewer fails due to java.lang.OutOfMemoryError exception.
    • hpcviewer writes a long list of Java error messages to the terminal!
    • hpcviewer attributes performance information only to functions and not to source code loops and lines! Why?
    • hpcviewer hangs trying to open a large database! Why?
    • hpcviewer runs glacially slowly! Why?
    • hpcviewer does not show my source code! Why?
      • Follow ‘best practices’
      • An explanation how HPCToolkit finds source files
    • hpcviewer’s reported line numbers do not exactly correspond to what I see in my source code! Why?
    • hpcviewer claims that there are several calls to a function within a particular source code scope, but my source code only has one! Why?
    • Trace view shows lots of white space on the left. Why?
  • Debugging
    • How do I debug HPCToolkit’s measurement?
    • Tracing libmonitor
    • Tracing HPCToolkit’s Measurement Subsystem
    • Using a debugger to inspect an execution being monitored by HPCToolkit
• Environment Variables
  • Environment Variables for Users
  • Environment Variables that May Avoid a Crash
  • Environment Variables for Developers
• References