虽然很早就接触了HPC，也参与过一些项目，诸如电影动画渲染集群以及某博导老师的基因分析计算集群，但是对于跑超算的linpack，一直没时间上手玩。
Linpack是超算必测项目,也是考验优化能力的套件,很有意思，记录下安装过程。
要正常运行，主要需要安装3个部分组件：
这三个组件，除了linpack本身是标准的外，MPI和BLAS都有多种版本不同的实现，各有千秋和应用场景或者优化的特点。当前主要为了跑通，所以mpi选用mipch2,BLAS选用openblas.
软件地址如下:
HPL - A Portable Implementation of the High-PerformanceLinpack Benchmark for Distributed-Memory Computers https://netlib.org/benchmark/hpl/
xianyi (Zhang Xianyi) · GitHubxianyi has 36 repositories available. Follow their code on GitHub. https://github.com/xianyi/openBLAS 直接下release里面最新的版本就可以。
Downloads | MPICH https://www.mpich.org/downloads/MPICH 直接下那个体积最大的包，编译完成后有Hydra这个程序管理可执行程序（mpiexec/mpirun）的。
先安装配置mpi，mpich2安装配置和其他开源软件没啥不同就是解压，configure make make install 几步
- tar zxf mpich-4.0.3.tar.gz
- cd mpich-4.0.3
- ./configure --prefix=/usr/local/mpich2/
- #需要提前安装好gfortran ,关于网上别的文章说的要添加--with-device=ch4:ofi 参数，我的configure的时候，是默认给我配置的这个，对于其他deivce类型，有需要优化的时候再慢慢看或者做测试吧。
- make -j4
- sudo make install
安装完成后需要配置环境变量,添加以下行变量到~/.bashrc末尾
- export PATH=$PATH:/usr/local/mpich2/bin/
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/mpich2/lib
- export MANPATH=$MANPATH:/usr/local/mpich2/share/man
然后拷贝/usr/local/mpich2整个目录到集群其他节点对应的位置，如果机器太多建议还是nfs之类的共享吧，否则某个软件改一下，所有机器又要重新拷贝一遍。
再然后就是配置ssh互信了。
- ssh-keygen -t rsa
- ssh-copy-id uos@192.168.5.249
- ssh-copy-id uos@192.168.5.230 #本机IP也要拷贝
- #完成拷贝后，直接ssh 192.168.5.249 或者 ssh 192.168.5.230验证看是否能免密登陆
-
配置完成后，跑个mpich2自带的跑圆周率计算的程序
- cd /home/uos/Downloads/mpich-4.0.3/examples
- #注意每个节点的这个mpich的源码业要拷贝过去。
- echo "192.168.5.230" >> nodes
- echo "192.168.5.249" >> nodes
- mpiexec -f nodes -n 2 ./cpi
下载的时候看了下，刚好这最后一版本还支持FT2000了，正好。
- tar zxf OpenBLAS-0.3.21.tar.gz
- cd OpenBLAS-0.3.21/
- sudo make PREFIX=/usr/local/openblas/ install
HPL的编译主要是mpi库和blas库的路径配置一下，设置下编译器，直接从解压的源码目录setup下面拷贝一个配置文件到源码根目录并改名为Make.FT2000-CBLAS做修改，这个名字后面编译需要用到，我修改后的配置如下：
- #
- # -- High Performance Computing Linpack Benchmark (HPL)
- # HPL - 2.3 - December 2, 2018
- # Antoine P. Petitet
- # University of Tennessee, Knoxville
- # Innovative Computing Laboratory
- # (C) Copyright 2000-2008 All Rights Reserved
- #
- # -- Copyright notice and Licensing terms:
- #
- # Redistribution and use in source and binary forms, with or without
- # modification, are permitted provided that the following conditions
- # are met:
- #
- # 1. Redistributions of source code must retain the above copyright
- # notice, this list of conditions and the following disclaimer.
- #
- # 2. Redistributions in binary form must reproduce the above copyright
- # notice, this list of conditions, and the following disclaimer in the
- # documentation and/or other materials provided with the distribution.
- #
- # 3. All advertising materials mentioning features or use of this
- # software must display the following acknowledgement:
- # This product includes software developed at the University of
- # Tennessee, Knoxville, Innovative Computing Laboratory.
- #
- # 4. The name of the University, the name of the Laboratory, or the
- # names of its contributors may not be used to endorse or promote
- # products derived from this software without specific written
- # permission.
- #
- # -- Disclaimer:
- #
- # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
- # OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- # DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- # ######################################################################
- #
- # ----------------------------------------------------------------------
- # - shell --------------------------------------------------------------
- # ----------------------------------------------------------------------
- #
- SHELL = /bin/sh
- #
- CD = cd
- CP = cp
- LN_S = ln -s
- MKDIR = mkdir
- RM = /bin/rm -f
- TOUCH = touch
- #
- # ----------------------------------------------------------------------
- # - Platform identifier ------------------------------------------------
- # ----------------------------------------------------------------------
- #
- ARCH = FT2000-CBLAS
- #
- # ----------------------------------------------------------------------
- # - HPL Directory Structure / HPL library ------------------------------
- # ----------------------------------------------------------------------
- #
- TOPdir = /home/uos/Downloads/hpl-2.3
- INCdir = $(TOPdir)/include
- BINdir = $(TOPdir)/bin/$(ARCH)
- LIBdir = $(TOPdir)/lib/$(ARCH)
- #
- HPLlib = $(LIBdir)/libhpl.a
- #
- # ----------------------------------------------------------------------
- # - Message Passing library (MPI) --------------------------------------
- # ----------------------------------------------------------------------
- # MPinc tells the C compiler where to find the Message Passing library
- # header files, MPlib is defined to be the name of the library to be
- # used. The variable MPdir is only used for defining MPinc and MPlib.
- #
- MPdir = /usr/local/mpich2
- MPinc = -I$(MPdir)/include
- MPlib = $(MPdir)/lib/libmpi.so
- #
- # ----------------------------------------------------------------------
- # - Linear Algebra library (BLAS or VSIPL) -----------------------------
- # ----------------------------------------------------------------------
- # LAinc tells the C compiler where to find the Linear Algebra library
- # header files, LAlib is defined to be the name of the library to be
- # used. The variable LAdir is only used for defining LAinc and LAlib.
- #
- LAdir = /usr/local/openblas
- LAinc = -I$(LAdir)/include
- LAlib = $(LAdir)/lib/libopenblas_ft2000p-r0.3.21.a
- #
- # ----------------------------------------------------------------------
- # - F77 / C interface --------------------------------------------------
- # ----------------------------------------------------------------------
- # You can skip this section if and only if you are not planning to use
- # a BLAS library featuring a Fortran 77 interface. Otherwise, it is
- # necessary to fill out the F2CDEFS variable with the appropriate
- # options. **One and only one** option should be chosen in **each** of
- # the 3 following categories:
- #
- # 1) name space (How C calls a Fortran 77 routine)
- #
- # -DAdd_ : all lower case and a suffixed underscore (Suns,
- # Intel, ...), [default]
- # -DNoChange : all lower case (IBM RS6000),
- # -DUpCase : all upper case (Cray),
- # -DAdd__ : the FORTRAN compiler in use is f2c.
- #
- # 2) C and Fortran 77 integer mapping
- #
- # -DF77_INTEGER=int : Fortran 77 INTEGER is a C int, [default]
- # -DF77_INTEGER=long : Fortran 77 INTEGER is a C long,
- # -DF77_INTEGER=short : Fortran 77 INTEGER is a C short.
- #
- # 3) Fortran 77 string handling
- #
- # -DStringSunStyle : The string address is passed at the string loca-
- # tion on the stack, and the string length is then
- # passed as an F77_INTEGER after all explicit
- # stack arguments, [default]
- # -DStringStructPtr : The address of a structure is passed by a
- # Fortran 77 string, and the structure is of the
- # form: struct {char *cp; F77_INTEGER len;},
- # -DStringStructVal : A structure is passed by value for each Fortran
- # 77 string, and the structure is of the form:
- # struct {char *cp; F77_INTEGER len;},
- # -DStringCrayStyle : Special option for Cray machines, which uses
- # Cray fcd (fortran character descriptor) for
- # interoperation.
- #
- F2CDEFS =
- #
- # ----------------------------------------------------------------------
- # - HPL includes / libraries / specifics -------------------------------
- # ----------------------------------------------------------------------
- #
- HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
- HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib)
- #
- # - Compile time options -----------------------------------------------
- #
- # -DHPL_COPY_L force the copy of the panel L before bcast;
- # -DHPL_CALL_CBLAS call the cblas interface;
- # -DHPL_CALL_VSIPL call the vsip library;
- # -DHPL_DETAILED_TIMING enable detailed timers;
- #
- # By default HPL will:
- # *) not copy L before broadcast,
- # *) call the BLAS Fortran 77 interface,
- # *) not display detailed timing information.
- #
- HPL_OPTS = -DHPL_CALL_CBLAS
- #
- # ----------------------------------------------------------------------
- #
- HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
- #
- # ----------------------------------------------------------------------
- # - Compilers / linkers - Optimization flags ---------------------------
- # ----------------------------------------------------------------------
- #
- CC = /usr/bin/gcc
- CCNOOPT = $(HPL_DEFS)
- CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops
- #
- # On some platforms, it is necessary to use the Fortran linker to find
- # the Fortran internals used in the BLAS library.
- #
- LINKER = /usr/bin/gfortran
- LINKFLAGS = $(CCFLAGS) -lpthread
- #
- ARCHIVER = ar
- ARFLAGS = r
- RANLIB = echo
- #
- # ----------------------------------------------------------------------
-
主要是修改TOPdirMPdirLAdirLINKERLINKFLAGS几个位置，编译的时候报错，LINKFLAGS 后面需要添加lpthread
修改完成后直接 make arch=FT2000-CBLAS ,编译完成后可执行程序在bin/FT2000-CBLAS/下，主要是xhplHPL.dat 两个文件。做测试的时候调整的参数都在HPL.dat 文件里面。
可以不用修改直接运行测试跑一下，
- cd bin/FT2000-CBLAS/
- mpirun -n 4 ./xhpl
如果没报错，说明安装正常。接下来绝大部分时间都需要慢慢调HPL.dat 里面的参数了，当然MPI以及BLAS库的优化业是必不可少的。
HPL Tuning https://netlib.org/benchmark/hpl/tuning.html
湘公网安备 43102202000103号
