Cray Programming Environment
The Cray Programming Environment (CPE) is a comprehensive suite of software tools, compilers, libraries, and debuggers designed to optimize and manage applications running on Cray supercomputers such as Komondor. The CPE is tailored to exploit the high-performance capabilities of Cray’s hardware, making it an essential toolkit for scientific computing, research, and other applications requiring massive computational power.
note:
Use CPE devices for maximum performance. CPE devices ensure maximum Slingshot optimized operation. Slingshot high speed network description: https://docs.hpc.kifu.hu/en/hardware/network.html
On Komondor, various compiler suites are available through module collections. These collections load the necessary modules, enabling users to work within one of the supported programming environments specific to Komondor.
Switching Compiler Suites
The compiler collections on Komondor are accessible through modules, and you can load them using the module load command:
module load PrgEnv-<name>
Here, <name> refers to the name of the desired compiler suite. Komondor offers the following main compiler collections, with the default being the Cray Compiling Environment (CCE).
Description |
Module Collection |
CCE - Cray Compiling Environment |
PrgEnv-cray |
GCC - GNU Compiler Collection |
PrgEnv-gnu |
Nvhpc - Nvidia Compiler |
PrgEnv-Nvhpc |
AOCC - AMD Optimizing C/C++ Compiler |
PrgEnv-aocc |
Intel - Intel Compiler |
PrgEnv-intel |
Example Usage
If you want to switch to the GNU Compiler Collection, you can use:
module load PrgEnv-gnu
Once you’ve loaded a programming environment, the compiler wrappers (cc, CC, and ftn) will be available.
Language |
Wrapper |
CCE |
GNU |
|---|---|---|---|
C |
man cc |
man CC |
man ftn |
C++ |
man craycc |
man crayCC |
man crayftn |
Fortran |
man gcc |
man g++ |
man gfortran |
The Cray Compiling Environment (CCE) module and its dependencies are loaded by default on each node and on the login server.
Currently Loaded Modulefiles:
1) cce/16.0.1 4) libfabric/1.15.2.0 7) cray-libsci/23.09.1.1
2) craype/2.7.23 5) craype-network-ofi 8) PrgEnv-cray/8.4.0
3) cray-dsmml/0.2.2 6) cray-mpich/8.1.27
The GNU compiler can be loaded instead of the CCE compiler:
module swap PrgEnv-cray PrgEnv-gnu
Useful Compiler Settings
Directive |
CCE Compiler’s Flag |
GNU Compiler’s Flag |
Nvidia Compiler’s Flag |
|---|---|---|---|
Default Optimization |
-O0 |
-O0 |
-O1 |
Aggressive Optimization |
-Ofast -flto |
-Ofast |
-O4 -fast |
Good Performance |
-O2 -funroll-loops -ffast-math |
-O2 -ftree-vectorize -funroll-loops -ffast-math |
|
A100 Card Optimization |
-target-accel=nvidia80 |
-gpu=cc80 |
|
Enable OpenMP |
-fopenmp |
-fopenmp |
-mp (CPU) -mp=gpu (GPU) |
Enable OpenAC |
-acc |
||
Debug |
-g -O0 |
-g -O0 |
-g -O0 (default) |
Verbose |
-v |
-v |
-v |
Note: The PrgEnv-aocc module is broken in versions 21.08 and 21.12.
Cray Scientific and Math Libraries (CSML)
CSML is a set of high performance libraries that provide portability for scientific applications by implementing APIs for arrays (NetCDF), sparse and dense linear algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW).
cray-libsci |
Cray Scientific Libraries |
cray-libsci_acc |
Cray Scientific Libraries for GPU |
cray-fftw |
Fastest Fourier Transform in the West (FFTW3) |
cray-parallel-netcdf |
Parallel I/O library for NetCDF file access |
cray-R |
R for use on HPE Cray HPC systems |
cray-python |
Python programming language and libraries for Cray PE |
Cray Message Passing Toolkit (CMPT)
CMPT is a collection of software libraries used to perform data transfers between nodes running in parallel applications. CMPT comprises the Message Passing Interface (MPI) and OpenSHMEM parallel programming models.
cray-mpich |
Cray MPICH Message Passing Interface |
cray-mpich-ucx |
Message Passing Interface (MPI) for the UCX netmod |
cray-openshmemx |
Logically shared distributed memory access routines |
cray-libpals |
Parallel Application Launch Service library |
cray-mpich-abi |
Cray MPICH ABI Compatibility module |
cray-mpich-abi-pre-intel-5.0 |
cray Cray MPICH pre-Intel MPI 5.0 ABI Compatibility module |
cray-pals |
Parallel Application Launch Service |
Performance Analysis and Optimization
cray-dsmml |
Distributed symmetric memory management library (DSMML) |
cray-pmi |
Cray Process Management Interface |
craype |
Setup for Cray PE driver set and targeting modules |
craype-dl-plugin-ftr |
Cray PE DL Plugin with fault tolerance support |
craype-dl-plugin-py3 |
Cray PE DL Plugin for accelerating distributed deep learning |
papi |
Performance API (PAPI) project specifies a standard API |
perftools-base |
Performance Tools module (CrayPat, Apprentice2, Reveal) |
Cray Debugger Support Tools (CDST)
gdb4hpc |
Cray Line Mode Parallel Debugger |
valgrind4hpc |
Valgrind-based debugging tool to aid in the detection of memory leaks |
cray-stat |
Cray Stack Trace Analysis Tool |
atp |
Abnormal Termination Processing (ATP) |
cray-ccdb |
Cray Comparative Debugger (CCDB) tool |
cray-mrnet |
Multicast Reduction Network module |
cray-cti |
Cray Common Tools Interface (CTI) |
cray-dyninst |
Dynamic instrumentation libraries |
sanitizers4hpc |
Fool for running HPC code instrumented with LLVM Sanitizer |
More information: Programming Models
Reference Manuals:
HPE Cray Programming Environment documentation: https://cpe.ext.hpe.com/docs/