OpenFOAM

OpenFOAM is a Finite Volume Method solver framework to solve CFD problems. Two important forks of OpenFOAM are supported:

Accessing OpenFOAM binaries @ Komondor

It is planned to compile the before mentioned OpenFOAM versions for various types of number representation.

The codes are placed in /opt/software/packages/openfoam/ folder. Recent versions using LMOD are placed in the /opt/software/packages/openfoam/LMOD/ folder.

Module use of OpenFOAM

There is an ongoing effort to make OpenFOAM available using ``module``s.

For experimenting you can try to make experimental module available:

module use /opt/software/packages/openfoam/modulefiles

and load one of them:

module load OpenFOAM/12

OpenFOAM.org version

It is intended to recompile even older versions since some recently recognized bugs are patched for older versions as well.

Version 12

The openfoam.org version 12 environment is configured using:

Clang compiled (using bfd linker) with -march=znver3 optimization flag:

module load PrgEnv-cray
source /opt/software/packages/openfoam/LMOD/OpenFOAM-12/etc/bashrc WM_COMPILER=Clang

Gcc compiled with -march=znver3 optimization flag:

module load PrgEnv-gnu
source /opt/software/packages/openfoam/LMOD/OpenFOAM-12/etc/bashrc WM_COMPILER=Gcc

For 64 bit version add also the WM_LABEL_SIZE=64 variable definition to the sourcing line:

module load PrgEnv-gnu
source /opt/software/packages/openfoam/LMOD/OpenFOAM-12/etc/bashrc WM_COMPILER=Gcc  WM_LABEL_SIZE=64

Version 11

At the time of writing the following commit is compiled: c219200

The openfoam.org version 11 enviroment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-11/etc/bashrc

You can select between various number representations (all influencing memory requirement and speed of your simulation) by:

For floating point numbers:

WM_PRECISION_OPTION=SP|DP|LP

(SP-single precision is usually not enough for convergence, DP-double is the optimal choice, LP-longdouble can be rearly required) LP have some MPI problem…

For integer numbers:

WM_LABEL_SIZE=32|64

64 bit version is needed if there are more than 2^31 (2 147 483 648) elements of separately defined mesh elements (e.g. points, faces, cells)

It is also useful to specify your working directory, since libraries will be compiled to that location.

WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-11/

<projectname> is the project name @ Komondor.

In summary you can access the required OpenFOAM executables e.g.:

source /opt/software/packages/openfoam/OpenFOAM-11/etc/bashrc WM_PRECISION_OPTION=DP WM_LABEL_SIZE=64  WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-11/

Version 10

The openfoam.org version 10 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-10/etc/bashrc

Only DP32 is available.

Version 9

The openfoam.org version 9 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-9/etc/bashrc

Only DP32 is available.

Since the number of files are strongly limited in your quota, it is strongly recommended to use the collated fileHandler, this is configured as default. If you have any anticipated problems, simple deactivate it for your actual case in system/controlDict:

OptimisationSwitches
{
    fileHandler uncollated;
}

Problems were found for snappyHexMesh-ing using the collated method. In this case -fileHandler uncollated should be used for these steps of the simulation and re- decompose it for the actual simulation.

OpenFOAM.com version

Version v2412

The openfoam.com version v2412 environment is configured using:

Gcc compiled:

module load PrgEnv-gnu cray-fftw gmp mpfr
source /opt/software/packages/openfoam/LMOD/OpenFOAM-v2412/etc/bashrc

Clang compiled (using lld linker):

Beside the novelties of v2412 version, these packages also contains CGAL dependent functions. Floating point precision is available in SPDP|DP, integer representation in 32|64 bit versions.

Version v2406

The openfoam.com version v2406 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-v2406/etc/bashrc

You can select between various number representations by (all influencing memory requirement and speed of your simulation):

For floating point numbers:

WM_PRECISION_OPTION=DP|SPDP

(DP-double is the usual choice, SPDP is a ‘recent’ development, a combination of single and double precision.)

For integer numbers:

WM_LABEL_SIZE=32|64

64 bit version is needed if there are more than 2^31 (2 147 483 648) elements of separtly defined mesh elements (e.g. points, faces, cells)

In summary you can access the required OpenFOAM executables e.g.:

source /opt/software/packages/openfoam/OpenFOAM-v2406/etc/bashrc WM_PRECISION_OPTION=SPDP WM_LABEL_SIZE=32 WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-v2406/

Version v2312

The openfoam.com version v2312 enviroment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-v2312/etc/bashrc

You can select between various number representations by (all influencing memory requirement and speed of your simulation):

For floating point numbers:

WM_PRECISION_OPTION=SP|DP|SPDP

(SP-single precision is usually not enough for convergence, DP-double is the usual choice, SPDP is a ‘recent’ development, a combination of single and double precision.)

For integer numbers:

WM_LABEL_SIZE=32|64

64 bit version is needed if there are more than 2^31 (2 147 483 648) elements of separately defined mesh elements (e.g. points, faces, cells)

In summary you can access the required OpenFOAM executables e.g.:

source /opt/software/packages/openfoam/OpenFOAM-v2312/etc/bashrc WM_PRECISION_OPTION=SPDP WM_LABEL_SIZE=32 WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-v2312/

Version v2306

The openfoam.com version v2302 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-v2306/etc/bashrc

Only DP32 is available.

Since the number of files are strongly limited in your quota, it is strongly recommended to use the collated fileHandler, this configured as default. If you have any problems simple deactivate it for your actual case in system/controlDict:

OptimisationSwitches
{
    fileHandler uncollated;
}

Compilation background

To compile @ Komondor following special settings are needed:

module load PrgEng-gnu crayx86-milan
WM_MPLIB=CRAY-MPICH

(For openfoam.org the CRAY-MPICH configuration is copied from the openfoam.com version)

The number presision is controlled by e.g.:

WM_PRECISION_OPTION=DP WM_LABEL_SIZE=32

Basic slurm settings

Simulation can be submitted using the following slurm script:

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --mail-type=END
#SBATCH --mail-user=your.name@mailserver.com

# SBATCH -c 8
# SBATCH -n 16
# SBATCH -N 1
# SBATCH --ntasks-per-node 4
#SBATCH --ntasks=256
#SBATCH -J motorBikeSteady_256
# SBATCH --exclusive
# SBATCH --mem-per-cpu=2000
#SBATCH --export=ALL

module load PrgEnv-gnu

source /opt/software/packages/openfoam/LMOD/OpenFOAM-12/etc/bashrc    WM_PRECISION_OPTION=DP WM_LABEL_SIZE=32

# To make number of cores passed to the domain decomposition
sed -i 's~numberOfSubdomains.*~numberOfSubdomains ${SLURM_NTASKS};~g'  system/decomposeParDict

./Allrun

Storage considerations

You should use the /project/ filesystem to store your case setup, than copy it to the /scratch/ filesystem and run it from there. When simulation converged, you should move your converged result back to your /project/ folder.

It is highly suggested to script this process!

Basic TODOs

It is useful to create your user directory:

mkdir -p /project/${project}/OpenFOAM/${USER}-12/run

Practical tools

Convergence plotting

Use gnuplot for convergence plotting, it is an efficient tool for continuous monitoring.

Editing

Use Visual Studio Code remote development technique for any longer editing.

OpenFOAM extension is useful for configuration file editing.

you should specify your username

username@komondor.hpc.kifu.hu

and also edit your ssh config file.

yourhomefolder/.ssh/config

Download results

It is efficient to use VS Code RightClick Download... option.

Terminal emulation

MobaXterm is a handy tool for terminal emulation.

Pre-processing tools

Salome-Platform is an opensource CAD and meshing framework. You can use it for automated/scripted geometry and mesh generation, in your OpenFOAM workflow.

Your generated salome script can be run using -t flag without GUI:

/opt/software/packages/openfoam/SALOME-9.14.0/salome -t script.py

if the above fails you can try the more tested version:

/opt/software/packages/openfoam/SALOME-9.12.0/salome -t script.py

Update: 10. March 2025.