File Management Considerations

Inode Limit

Working with a lot of small files, either they are data files (e.g. temporary outputs) or software components (e.g. libraries or packages), can have a huge negative impact on the performance of the Lustre file system and also easily result in running out of inode (number of files) limits.

Here are some tips, that can help to overcome this limitation. Whether these are appropriate and effective depends on the specific application and use case:

  • First of all, try to optimize your workflow to avoid generating or processing a lot of files at once. For example, you can pack temporary data that is not currently being used into a tar archive.

  • Use a singularity container to store your files. This method is particularly suitable e.g. for installing conda packages with many dependencies. Examples: Copying packed data into the container, Python environment in the container.

  • Consider implementing HDF5 data structure in your application. HDF5 provides good parallel performance, even for writing under certain conditions. Example: An MNIST example with Pytorch and HDF5.

  • Put the files into a single filesystem image file and mount it into a directory using FUSE. You can also try fuse-zip to directly mount zip archives (type man fuse-zip in the command line for help).

FUSE

FUSE (Filesystem in Userspace) can help you save Inodes in the filesystem.

Pros

  • store multiple files on one Inode

  • access times for small files may (although not necessarily) improve

Cons

  • read/write operations have additional overhead, which can lead to significant overall slowdowns

  • it is writeable from only one node at a time (on other nodes it is read-only)

  • if content is modified then you have to re-mount on all other nodes

  • size of the image is predefined and space is allocated at creation

Create an image file with proper Lustre striping (i.e. over how many OSTs - object storage targets - should the image file be striped; see: Lustre performance considerations

lfs setstripe -c 2 ./store.img

Grow the image to have a size of 100 GB:

truncate --size=100G store.img

Create an EXT4 filesystem on the image:

mkfs.ext4 store.img

Note

If working with many small files, consider creating the filesystem with the -T small option.

Create a directory which will act as a mount point for the image:

mkdir store

Mounting for RW (read/write):

fuse2fs -o fakeroot,auto_unmount store.img store

Mounting for RO (read-only):

fuse2fs -o fakeroot,auto_unmount,ro store.img store

Unmount:

fusermount -u store

Checking available space / Inodes:

$ df -h store
Filesystem      Size  Used Avail Use% Mounted on
store.img        99G  573M   93G   1% /project/home/username/store

$ df -i store
Filesystem      Inodes IUsed   IFree IUse% Mounted on
store.img      6553600    11 6553589    1% /project/home/user/store