Q-Chem & BrianQC

Quick start guide for Q-Chem + BrianQC on HPC komondor

Prerequisites for Q-Chem & BrianQC

In order to start a job with either of Q-Chem, BrianQC or Q-Chem+BrianQC on HPC Komondor you need to have your login credentials, and have a positive balance (sbalance command).

accessing Komondor see: https://docs.hpc.kifu.hu/first-steps/index.html
signing in to Komondor see: https://docs.hpc.kifu.hu/first-steps/ssh-key.html
requesting projects and HPC processor and GPU time see: https://docs.hpc.kifu.hu/first-steps/projects.html
Allocating resource on HPC, salloc, srun, sbatch see: https://slurm.schedmd.com/tutorials.html

For Q-Chem manual/documentation, please visit: https://manual.q-chem.com/latest/

For BrianQC manual please visit: https://streamnovation-brianqc-manual.s3.amazonaws.com/BrianQC_Manual_v1_3_1.pdf

For more information on Q-Chem please visit: https://www.q-chem.com/

For more information on BrianQC see: https://www.brianqc.com/

Starting Q-Chem or Q-Chem+BrianQC

Allocate somehow a resource: salloc, srun or sbatch. For example

salloc -c 10 --mem-per-cpu=1900 -p ai --gpus=1
srun --jobid=<Allocated JOBID> --pty bash -i

will allocate a job on the ai partition with a single gpu and 10 CPU cores with 19 GB of memory and start an interactive shell session. For sample sbatch script see section Example.

This ‘source’ command is needed once per session:

source /opt/software/packages/qchem/6.1.0/qcenv.sh

This commands starts Q-Chem on 32 CPU threads:

qchem -nt 32 ~/user-inputs/calculation1.in

This commands starts Q-Chem on 16 CPU threads and on the available GPUs:

qchem -nt 16 -gpu ~/user-inputs/calculation1.in

BrianQC by default will use as many GPUs as possible during an execution. As a result if no environment variables of the above are given BrianQC will use all devices in your system allowed by the license. There are more ways to control the number of GPUs used for calculations:

CUDA_VISIBLE_DEVICES environment variable,
BRIANQC_GPU_COUNT environment variable

For more info see section 5.3 ‘Handling multiple GPUs’ in the BrianQC manual.

Examples

Create a script file: touch run_qchem.sh Edit the file to have the following content to run a Q-Chem+BrianQC calculation:

#!/bin/bash
#SBATCH -A <YOUR KOMONDOR ACCOUNT NAME, NOT USER NAME>
#SBATCH --job-name=<QCHEM_TEST_XX>
#SBATCH --time=1-02:03:04
#SBATCH --nodes=1
#SBATCH --partition=gpu
#SBATCH --gpus=2
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=1900M

source /opt/software/packages/qchem/6.1.0/qcenv.sh
qchem -nt 16 -gpu ~/user-inputs/task_to_calculate.in > ~/user-inputs/task_to_calculate.out 2>&1

This will start a batch job on the gpu partition of komondor, allocating part of a single node with: 16 cores of CPU, 2 GPUs, and 30.4 GB of memory. Source the environment variables for Q-Chem and start a Q-Chem job with 16 threads and gpu support.

Edit the file to have the following content to run a Q-Chem only calculation:

#!/bin/bash
#SBATCH -A <YOUR KOMONDOR ACCOUNT NAME, NOT USER NAME>
#SBATCH --job-name=<QCHEM_TEST_XX>
#SBATCH --time=2-01:01:07
#SBATCH --nodes=1
#SBATCH --partition=cpu
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=1900M

source /opt/software/packages/qchem/6.1.0/qcenv.sh
qchem -nt 16  ~/user-inputs/task_to_calculate.in > ~/user-inputs/task_to_calculate.out 2>&1

See difference in partition that is cpu and not gpu and the missing -gpu command line option during invoking Q-Chem binary.

To run this script make it executable: chmod +x run_qchem.sh Send it to the job scheduler: sbatch run_qchem.sh

The job will be covered from the specified komondor account balance. The result will be written to the ~/user-inputs/task_to_calculate.out text file.