OpenFOAM @ Komondor

OpenFOAM is a Finite Volume Method solver framework to solve CFD problems. Two important forks of OpenFOAM are supported:

• openfoam.org

• openfoam.com

Accessing OpenFOAM binaries @ Komondor

It is planned to compile the before mentioned OpenFOAM versions for various types of number representation.

The codes are placed in /opt/software/packages/openfoam/ folder.

OpenFOAM.org version

It is intended to recompile even older versions since some recently recognised bugs are patched for older versions as well.

Version 11

At the time of writing the following commit is compiled: c219200

The openfoam.org version 11 enviroment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-11/etc/bashrc

You can select between various number representations (all influencing memory requirement and speed of your simulation) by:

For floating point numbers:

WM_PRECISION_OPTION=SP|DP|LP

(SP-single precision is usually not enough for convergence, DP-double is the optimal choice, LP-longdouble can be rearly required) LP have some MPI problem…

For integer numbers:

WM_LABEL_SIZE=32|64

64 bit version is needed if there are more than 2^31 (2 147 483 648) elements of separtly defined mesh elements (e.g. points, faces, cells)

It is also useful to specify your working directory, since libraries will be compiled to that location.

WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-11/

<projectname> is the project name @ Komondor.

In summary you can access the required OpenFOAM executables e.g.:

source /opt/software/packages/openfoam/OpenFOAM-11/etc/bashrc WM_PRECISION_OPTION=DP WM_LABEL_SIZE=64  WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-11/

Version 10

The openfoam.org version 10 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-10/etc/bashrc

Only DP32 is available.

Version 9

The openfoam.org version 9 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-9/etc/bashrc

Only DP32 is available.

OpenFOAM.org Parallel I/O

Since the number of files are strongly limited in your quota, it is strongly recommended to use the collated fileHandler, this is configured as default. If you have any anticipated problems, simple deactivate it for your actual case in system/controlDict:

OptimisationSwitches
{
    fileHandler uncollated;
}

Problems were found for snappyHexMesh-ing using the collated method. In this case -fileHandler uncollated should be used for these steps of the simulation and re- decompose it for the actual simulation.

OpenFOAM.com version

Version v2312

The openfoam.com version v2312 enviroment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-v2312/etc/bashrc

You can select between various number representations by (all influencing memory requirement and speed of your simulation):

For floating point numbers:

WM_PRECISION_OPTION=SP|DP|SPDP

(SP-single precision is usually not enough for convergence, DP-double is the usual choice, SPDP is a ‘recent’ development, a combination of single and double precison.)

For integer numbers:

WM_LABEL_SIZE=32|64

64 bit version is needed if there are more than 2^31 (2 147 483 648) elements of separtly defined mesh elements (e.g. points, faces, cells)

In summary you can acces the required OpenFOAM executables e.g.:

source /opt/software/packages/openfoam/OpenFOAM-v2312/etc/bashrc WM_PRECISION_OPTION=SPDP WM_LABEL_SIZE=32 WM_PROJECT_USER_DIR=/project/<projectname>/OpenFOAM/${USER}-v2312/

Version v2306

The openfoam.com version v2302 environment is configured using:

source /opt/software/packages/openfoam/OpenFOAM-v2306/etc/bashrc

Only DP32 is available.

OpenFOAM.com Parallel I/O

Since the number of files are strongly limited in your quota, it is strongly recommended to use the collated fileHandler, this configured as default. If you have any problems simple deactivate it for your actual case in system/controlDict:

OptimisationSwitches
{
    fileHandler uncollated;
}

Compilation background

To compile @ Komondor following special settings are needed:

module load PrgEng-gnu crayx86-milan
WM_MPLIB=CRAY-MPICH

(For openfoam.org the CRAY-MPICH configuration is copied from the openfoam.com version)

The number presision is controlled by e.g.:

WM_PRECISION_OPTION=DP WM_LABEL_SIZE=32

Basic slurm settings

Simulation can be submitted using the following slurm script:

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --mail-type=END
#SBATCH --mail-user=your.name@mailserver.com

# SBATCH -c 8
# SBATCH -n 16
# SBATCH -N 1
# SBATCH --ntasks-per-node 4
#SBATCH --ntasks=256
#SBATCH -J motorBikeSteady_256
#SBATCH --exclusive
# SBATCH --mem-per-cpu=2000
# SBATCH --mem-per-cpu=MEMORY
# SBATCH --memory=0
#SBATCH --export=ALL

module purge
module load PrgEnv-cray craype-x86-milan



source /opt/software/packages/openfoam/OpenFOAM-11/etc/bashrc    WM_PRECISION_OPTION=DP WM_LABEL_SIZE=32

# To make number of cores passed to the domain decomposition
sed -i 's~numberOfSubdomains.*~numberOfSubdomains ${SLURM_NTASKS};~g'  system/decomposeParDict

./Allrun

Storage considerations

You should use the /project/ filesystem to store your case setup, than copy it to the /scratch/ filesystem and run it from there. When simulation converged, you should move your converged result back to your /project/ folder.

It is highly suggested to script this process!

Basic TODOs

It is useful to create your user directory:

mkdir -p /project/${project}/OpenFOAM/${USER}-11/run

Practical tools

Convergence plotting

Use gnuplot for convergence plotting, it is an efficient tool for continous monitoring.

Editing

Use Visual Studio Code remote development technique for any longer editing.

OpenFOAM extension is useful for configuration file editing.

you should specify your username

username@komondor.hpc.kifu.hu

and also edit your ssh config file.

yourhomefolder/.ssh/config

Download results

It is efficient to use VS Code RightClick Download... option.

Terminal emulation

MobaXterm is a handy tool for terminal emulation.

Pre-processing tools

Salome-Platform

Salome-Platform is an opensource CAD and meshing framework. You can use it for automated/scripted geometry and mesh generation, in your OpenFOAM workflow.

Your generated salome script can be run using -t flag without GUI:

/opt/software/packages/openfoam/SALOME-9.12.0/salome -t script.py