容器的md5值:6db7c30cc8844ee5230bd9880e722bc0 使用方法： (注：“/path/to”表示举例路径，现网需要根据实际情况调整，建议部署在共享路径中，后续章节凡是遇到安装路径的命令，都以现网实际规划的安装路径为准进行替换，不再单独说明。) 一、安装singularity yum -y install libatomic libstdc++ libstdc++-devel libseccomp-devel glib2-devel gcc squashfs-tools tar cd /path/to/go_build  wget https://go.dev/dl/go1.19.linux-arm64.tar.gz --no-ch  tar -C /path/to/go_install -xf go1.19.linux-arm64.tar.gz  export PATH=/path/to/go_install/go/bin:$PATH source ~/.bashrc cd /path/to/singularity_build  wget https://github.com/sylabs/singularity/releases/download/v3.10.2/singularity-ce-3.10.2.tar.gz --no-ch  tar -xf singularity-ce-3.10.2.tar.gz  cd singularity-ce-3.10.2  ./mconfig --prefix=/path/to/singularity_install make -C ./builddir  make -C ./builddir install  export PATH=/path/to/singularity_install/bin:$PATH 二、安装hmpi cd /path/to/bs_build wget https://mirrors.huaweicloud.com/kunpeng/archive/compiler/bisheng_compiler/BiShengCompiler-3.2.0-aarch64-linux-mini.tar.gz tar xzvf BiShengCompiler-3.2.0-aarch64-linux-mini.tar.gz -C /path/to/bs_install --strip-components=1 export PATH=/path/to/bs_install/bin:$PATH export LD_LIBRARY_PATH=/path/to/bs_install/lib:/path/to/bs_install/lib/clang/15.0.4/lib:$LD_LIBRARY_PATH export INCLUDE=/path/to/bs_install/include:/path/to/bs_install/lib/clang/15.0.4/include:$INCLUDE export CC=clang CXX=clang++ FC=flang yum install -y perl-Data-Dumper autoconf automake libtool binutils flex cmake cd /path/to/hucx_build wget https://gitee.com/kunpengcompute/hucx/repository/archive/v2.3.0-huawei.tar.gz -f hucx-2.3.0-huawei.tar.gz tar xf hucx-2.3.0-huawei.tar.gz cd hucx-v2.3.0-huawei ./autogen.sh ./contrib/configure-opt --prefix=/path/to/hucx_install --disable-numa --enable-mt CFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" CXXFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" make -j make -j install export PATH=/path/to/hucx_install/bin:$PATH export LD_LIBRARY_PATH=/path/to/hucx_install/lib:$LD_LIBRARY_PATH export C_INCLUDE_PATH=/path/to/hucx_install/include:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/path/to/hucx_install/include:$CPLUS_INCLUDE_PATH export CFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" CXXFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" FFLAGS="-Wno-error=int-conversion" cd /path/to/xucg_build wget https://gitee.com/kunpengcompute/xucg/repository/archive/v2.3.0-huawei.tar.gz -f xucg-2.3.0-huawei.tar.gz tar xf xucg-2.3.0-huawei.tar.gz cd xucg-v2.3.0-huawei mkdir build && cd build cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/xucg_install -DCMAKE_BUILD_TYPE=Release -DUCG_BUILD_WITH_UCX=/path/to/hucx_install -DUCG_ENABLE_MT=ON -DUCG_BUILD_TESTS=OFF -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_Fortran_COMPILER=$FC make -j make -j install export PATH=/path/to/xucg_install/bin:$PATH export LD_LIBRARY_PATH=/path/to/xucg_install/lib:$LD_LIBRARY_PATH export C_INCLUDE_PATH=/path/to/xucg_install/include:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/path/to/xucg_install/include:$CPLUS_INCLUDE_PATH cd /path/to/hmpi_build wget https://gitee.com/kunpengcompute/hmpi/repository/archive/v2.3.0-huawei.tar.gz -f hmpi-2.3.0-huawei.tar.gz tar xf hmpi-2.3.0-huawei.tar.gz cd hmpi-v2.3.0-huawei/ ./autogen.pl ./configure --prefix=/path/to/hmpi_install --with-platform=contrib/platform/mellanox/optimized --enable-mpi1-compatibility --with-ucx=/path/to/hucx_install --with-ucg=/path/to/xucg_install make -j make -j install export PATH=/path/to/hmpi_install/bin:$PATH export LD_LIBRARY_PATH=/path/to/hmpi_install/lib:$LD_LIBRARY_PATH export C_INCLUDE_PATH=/path/to/hmpi_install/include:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/path/to/hmpi_install/include:$CPLUS_INCLUDE_PATH 三、获取容器sif 在本页面下载openfoam1906-SVE_v1.sif文件，并上传到服务器目录/path/to/sif 四、获取测试算例case 通过该链接下载算例https://sourceforge.net/projects/openfoam/files/v1906/OpenFOAM-v1906.tgz 测试算例包含在源码目录中 五、运行容器 1、单节点运行(使用容器内部mpi运行） 1.1 根据实际需要设置网格精度 cd /path/to/OpenFOAM/tutorials/incompressible/pisoFoam/LES/motorBike/motorBike sed -ri "s/\(20 8 8\)/\(2048 512 64\)/" ./system/blockMeshDict 1.2 执行以下命令设置运行CPU核数。 sed -ri "s/8/128/" system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch} sed -ri "s/\(4 2 1\)/\(16 4 2\)/" system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch} （说明：此处配置为单节点运行，运行CPU核数为128，16*4*2=128。可根据具体情况修改。） 1.3 编写执行脚本 export WM_PROJECT_DIR= #解压源码路径 export TEST_DIR= #motorBike路径 export SIF_DIR= #镜像路径 export NP= #总核数 singularity exec -B $TEST_DIR $SIF_DIR $TEST_DIR/Allclean time -p singularity exec -B $TEST_DIR $SIF_DIR $TEST_DIR/Allrun （说明该方式是使用容器内部的hmpi,不依赖物理机上的mpi直接运行，对不需要进行多节点运行的用户可选择命令2，即不需要 额外安装mpi） 2、单/多节点运行 2.1 根据实际需要设置网格精度 cd /path/to/OpenFOAM/tutorials/incompressible/pisoFoam/LES/motorBike/motorBike sed -ri "s/\(20 8 8\)/\(2048 512 64\)/" ./system/blockMeshDict 2.2 执行以下命令设置运行CPU核数。 sed -ri "s/8/256/" system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch} sed -ri "s/\(4 2 1\)/\(16 4 4\)/" system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch} （说明：此处配置为双节点运行，运行CPU核数为256，16*4*4=256。可根据具体情况修改。） 2.3 编写执行脚本 export WM_PROJECT_DIR= #解压源码路径 export TEST_DIR= #motorBike路径 export SIF_DIR= #镜像路径 export NP= #总核数 cd $TEST_DIR rm -rf processor* rm -rf log* \cp $WM_PROJECT_DIR/tutorials/resources/geometry/motorBike.obj.gz constant/triSurface/ # Make dummy 0 directory rm 0 -rf mkdir 0 #sed -ri "s/\(20 8 8\)/\(2048 512 64\)/" ./system/blockMeshDict #sed -ri "s/8/256/" system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch} #sed -ri "s/\(4 2 1\)/\(16 4 4\)/" system/{decomposeParDict,decomposeParDict.hierarchical,decomposeParDict.ptscotch} restore0Dir() { case "$1" in -processor | -processors) echo "Restore 0/ from 0.orig/ for processor directories" [ -d 0.orig ] || echo " Warning: no 0.orig/ found" # do nonetheless #printf "===================================" echo "=====================================" \ls -d processor* | xargs -I {} \rm -rf ./{}/0 \ls -d processor* | xargs -I {} \cp -r 0.orig ./{}/0 > /dev/null 2>&1 # Remove '#include' directives from field dictionaries # for collated format if [ "$1" = "-processors" ] then ( echo "Filter #include directives in processors/0:" \cd processors/0 2>/dev/null || exit 0 for file in $(grep -l "#include" * 2> /dev/null) do singularity exec -B $WM_PROJECT_DIR $SIF_DIR foamDictionary "$file" > "$file.$$." && mv "$file.$$." "$file" echo " $file" done | tr -d '\n' echo ) fi ;; *) echo "Restore 0/ from 0.orig/" if [ -d 0.orig ] then \rm -rf 0 \cp -r 0.orig 0 2>/dev/null else echo " Warning: no 0.orig/ found" fi ;; esac } #RunAPP singularity exec -B $WM_PROJECT_DIR $SIF_DIR blockMesh > log.blockMesh 2>&1 singularity exec -B $WM_PROJECT_DIR $SIF_DIR decomposePar -decomposeParDict system/decomposeParDict.hierarchical > log.decomposePar 2>&1 #Runparallel run -np $NP -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x WM_PROJECT_DIR=$WM_PROJECT_DIR -hostfile $HOSTFILE singularity exec -B $WM_PROJECT_DIR $SIF_DIR snappyHexMesh -parallel -decomposeParDict system/decomposeParDict.ptscotch -profiling -overwrite > log.snappyHexMesh 2>&1 find . -iname '*level*' -type f -delete restore0Dir -processor mpirun -np $NP -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x WM_PROJECT_DIR=$WM_PROJECT_DIR -hostfile $HOSTFILE singularity exec -B $WM_PROJECT_DIR $SIF_DIR renumberMesh -parallel -overwrite > log.renumberMesh 2>&1 mpirun -np $NP -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x WM_PROJECT_DIR=$WM_PROJECT_DIR -hostfile $HOSTFILE singularity exec -B $WM_PROJECT_DIR $SIF_DIR potentialFoam -parallel -initialiseUBCs > log.potentialFoam 2>&1 mpirun -np $NP -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x WM_PROJECT_DIR=$WM_PROJECT_DIR -hostfile $HOSTFILE singularity exec -B $WM_PROJECT_DIR $SIF_DIR checkMesh -parallel -writeFields '(nonOrthoAngle)' -constant > log.checkMesh 2>&1 mpirun -np $NP -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x WM_PROJECT_DIR=$WM_PROJECT_DIR -hostfile $HOSTFILE singularity exec -B $WM_PROJECT_DIR $SIF_DIR simpleFoam -parallel > log.simpleFoam 2>&1 （说明：-B：绑定本地算例目录，exec命令进入容器为只读模式，不可在容器内对文件进行读写操作。np为MPI进程数量，如果没有开启超线程技术，MPI进程数乘以OMP线程数应该小于等于CPU核数。程序运行结束时，在当前目录生成多个log，需要查看日志文件log.simpleFoam中的“ExecutionTime”值，单位是“s”，数值越小性能越优。）