Cray Programming Environment

The Cray Programming Environment (CPE) is a comprehensive suite of software tools, compilers, libraries, and debuggers designed to optimize and manage applications running on Cray supercomputers such as Komondor. The CPE is tailored to exploit the high-performance capabilities of Cray’s hardware, making it an essential toolkit for scientific computing, research, and other applications requiring massive computational power.

note:

Use CPE devices for maximum performance. CPE devices ensure maximum Slingshot optimized operation. Slingshot high speed network description: https://docs.hpc.kifu.hu/en/hardware/network.html

On Komondor, various compiler suites are available through module collections. These collections load the necessary modules, enabling users to work within one of the supported programming environments specific to Komondor.

Switching Compiler Suites

The compiler collections on Komondor are accessible through modules, and you can load them using the module load command:

module load PrgEnv-<name>

Here, <name> refers to the name of the desired compiler suite. Komondor offers the following main compiler collections, with the default being the Cray Compiling Environment (CCE).

Description

Module Collection

CCE - Cray Compiling Environment

PrgEnv-cray

GCC - GNU Compiler Collection

PrgEnv-gnu

Nvhpc - Nvidia Compiler

PrgEnv-Nvhpc

AOCC - AMD Optimizing C/C++ Compiler

PrgEnv-aocc

Intel - Intel Compiler

PrgEnv-intel

Example Usage

If you want to switch to the GNU Compiler Collection, you can use:

module load PrgEnv-gnu

Once you’ve loaded a programming environment, the compiler wrappers (cc, CC, and ftn) will be available.

Language

Wrapper

CCE

GNU

C

man cc

man CC

man ftn

C++

man craycc

man crayCC

man crayftn

Fortran

man gcc

man g++

man gfortran

The Cray Compiling Environment (CCE) module and its dependencies are loaded by default on each node and on the login server.

Currently Loaded Modulefiles:
1) cce/16.0.1              4) libfabric/1.15.2.0      7) cray-libsci/23.09.1.1
2) craype/2.7.23           5) craype-network-ofi      8) PrgEnv-cray/8.4.0
3) cray-dsmml/0.2.2        6) cray-mpich/8.1.27

The GNU compiler can be loaded instead of the CCE compiler:

module swap PrgEnv-cray PrgEnv-gnu

Useful Compiler Settings

Directive

CCE Compiler’s Flag

GNU Compiler’s Flag

Nvidia Compiler’s Flag

Default Optimization

-O0

-O0

-O1

Aggressive Optimization

-Ofast -flto

-Ofast

-O4 -fast

Good Performance

-O2 -funroll-loops -ffast-math

-O2 -ftree-vectorize -funroll-loops -ffast-math

A100 Card Optimization

-target-accel=nvidia80

-gpu=cc80

Enable OpenMP

-fopenmp

-fopenmp

-mp (CPU) -mp=gpu (GPU)

Enable OpenAC

-acc

Debug

-g -O0

-g -O0

-g -O0 (default)

Verbose

-v

-v

-v

Note: The PrgEnv-aocc module is broken in versions 21.08 and 21.12.

Cray Scientific and Math Libraries (CSML)

CSML is a set of high performance libraries that provide portability for scientific applications by implementing APIs for arrays (NetCDF), sparse and dense linear algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW).

cray-libsci

Cray Scientific Libraries

cray-libsci_acc

Cray Scientific Libraries for GPU

cray-fftw

Fastest Fourier Transform in the West (FFTW3)

cray-parallel-netcdf

Parallel I/O library for NetCDF file access

cray-R

R for use on HPE Cray HPC systems

cray-python

Python programming language and libraries for Cray PE

Cray Message Passing Toolkit (CMPT)

CMPT is a collection of software libraries used to perform data transfers between nodes running in parallel applications. CMPT comprises the Message Passing Interface (MPI) and OpenSHMEM parallel programming models.

cray-mpich

Cray MPICH Message Passing Interface

cray-mpich-ucx

Message Passing Interface (MPI) for the UCX netmod

cray-openshmemx

Logically shared distributed memory access routines

cray-libpals

Parallel Application Launch Service library

cray-mpich-abi

Cray MPICH ABI Compatibility module

cray-mpich-abi-pre-intel-5.0

cray Cray MPICH pre-Intel MPI 5.0 ABI Compatibility module

cray-pals

Parallel Application Launch Service

Performance Analysis and Optimization

cray-dsmml

Distributed symmetric memory management library (DSMML)

cray-pmi

Cray Process Management Interface

craype

Setup for Cray PE driver set and targeting modules

craype-dl-plugin-ftr

Cray PE DL Plugin with fault tolerance support

craype-dl-plugin-py3

Cray PE DL Plugin for accelerating distributed deep learning

papi

Performance API (PAPI) project specifies a standard API

perftools-base

Performance Tools module (CrayPat, Apprentice2, Reveal)

Cray Debugger Support Tools (CDST)

gdb4hpc

Cray Line Mode Parallel Debugger

valgrind4hpc

Valgrind-based debugging tool to aid in the detection of memory leaks

cray-stat

Cray Stack Trace Analysis Tool

atp

Abnormal Termination Processing (ATP)

cray-ccdb

Cray Comparative Debugger (CCDB) tool

cray-mrnet

Multicast Reduction Network module

cray-cti

Cray Common Tools Interface (CTI)

cray-dyninst

Dynamic instrumentation libraries

sanitizers4hpc

Fool for running HPC code instrumented with LLVM Sanitizer

More information: Programming Models

Reference Manuals:

HPE Cray Programming Environment documentation: https://cpe.ext.hpe.com/docs/