如何强制打开mpi 3使用TCP?

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了如何强制打开mpi 3使用TCP?相关的知识,希望对你有一定的参考价值。

我运行了一个小型集群进行mpi计算,最近我们购买了一些EDR Infiniband设备。我正在用两台计算机进行测试,通过一个非托管交换机连接,我能够在两个节点中运行一个有30个进程的测试程序。监控Infiband计数器在

/sys/class/infiniband/mlx5_0/ports/1/counters

现在我想用以太网TCP连接运行同样的测试,以获得一些关于Inifiniband提供的真正改进的数据,但我无法使其工作。我已经尝试了所有类型的mca参数,如。

/usr/lib64/openmpi3/bin/mpirun --hostfile hostfile.txt --mca btl ^openib -np 30 nek5000
/usr/lib64/openmpi3/bin/mpirun --hostfile hostfile.txt --mca btl tcp,self,vader -np 30 nek5000
/usr/lib64/openmpi3/bin/mpirun --hostfile hostfile.txt --mca btl tcp,self,vader --mca btl_openib_if_exclude mlx5_0  -np 30 nek5000

都没有成功 当程序运行时,Infiniband的计数器在增加,运行时间与Infiniband的没有区别。两台电脑都运行Centos 7,这是ompi_info的结果。

                 Package: Open MPI mockbuild@x86-02.bsys.centos.org
                          Distribution
                Open MPI: 3.1.3
  Open MPI repo revision: v3.1.3
   Open MPI release date: Oct 29, 2018
                Open RTE: 3.1.3
  Open RTE repo revision: v3.1.3
   Open RTE release date: Oct 29, 2018
                    OPAL: 3.1.3
      OPAL repo revision: v3.1.3
       OPAL release date: Oct 29, 2018
                 MPI API: 3.1.0
            Ident string: 3.1.3
                  Prefix: /usr/lib64/openmpi3
 Configured architecture: x86_64-unknown-linux-gnu
          Configure host: x86-02.bsys.centos.org
           Configured by: mockbuild
           Configured on: Thu Aug 22 16:51:48 UTC 2019
          Configure host: x86-02.bsys.centos.org
  Configure command line: '--prefix=/usr/lib64/openmpi3'
                          '--mandir=/usr/share/man/openmpi3-x86_64'
                          '--includedir=/usr/include/openmpi3-x86_64'
                          '--sysconfdir=/etc/openmpi3-x86_64'
                          '--disable-silent-rules' '--enable-builtin-atomics'
                          '--enable-mpi-cxx' '--with-sge' '--with-valgrind'
                          '--enable-memchecker' '--with-hwloc=/usr'
                          '--with-ucx' 'CC=gcc' 'CXX=g++'
                          'LDFLAGS=-Wl,-z,relro ' 'CFLAGS= -O2 -g -pipe -Wall
                          -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
                          -fstack-protector-strong --param=ssp-buffer-size=4
                          -grecord-gcc-switches   -m64 -mtune=generic'
                          'CXXFLAGS= -O2 -g -pipe -Wall
                          -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
                          -fstack-protector-strong --param=ssp-buffer-size=4
                          -grecord-gcc-switches   -m64 -mtune=generic'
                          'FC=gfortran' 'FCFLAGS= -O2 -g -pipe -Wall
                          -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
                          -fstack-protector-strong --param=ssp-buffer-size=4
                          -grecord-gcc-switches   -m64 -mtune=generic'
                Built by: mockbuild
                Built on: Thu Aug 22 16:55:56 UTC 2019
              Built host: x86-02.bsys.centos.org
              C bindings: yes
            C++ bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (limited: overloading)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: no
 Fort mpi_f08 compliance: The mpi_f08 module was not built
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: runpath
              C compiler: gcc
     C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
      C compiler version: 4.8.5
            C++ compiler: g++
   C++ compiler absolute: /usr/bin/g++
           Fort compiler: gfortran
       Fort compiler abs: /usr/bin/gfortran
         Fort ignore TKR: no
   Fort 08 assumed shape: no
      Fort optional args: no
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: no
      Fort BIND(C) (all): no
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): no
       Fort TYPE,BIND(C): no
 Fort T,BIND(C,name="a"): no
            Fort PRIVATE: no
          Fort PROTECTED: no
           Fort ABSTRACT: no
       Fort ASYNCHRONOUS: no
          Fort PROCEDURE: no
         Fort USE...ONLY: no
           Fort C_FUNLOC: no
 Fort f08 using wrappers: no
         Fort MPI_SIZEOF: no
             C profiling: yes
           C++ profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: no
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, ORTE progress: yes, Event lib:
                          yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
          MPI extensions: affinity, cuda
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v3.1.3)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v3.1.3)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA btl: self (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA btl: usnic (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v3.1.3)
            MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v3.1.3)
            MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA crs: none (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
               MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v3.1.3)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v3.1.3)
          MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v3.1.3)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v3.1.3)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v3.1.3)
                MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA pmix: pmix2x (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v3.1.3)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v3.1.3)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v3.1.3)
                 MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v3.1.3)
                 MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v3.1.3)
              MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
                          v3.1.3)
              MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
                          v3.1.3)
              MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
                          v3.1.3)
              MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
                          v3.1.3)
              MCA errmgr: dvm (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA ess: env (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
                          v3.1.3)
                 MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v3.1.3)
               MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v3.1.3)
             MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v3.1.3)
            MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v3.1.3)
                MCA odls: default (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA oob: ud (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v3.1.3)
                MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
               MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
               MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
               MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA rml: ofi (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v3.1.3)
              MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v3.1.3)
              MCA routed: debruijn (MCA v2.1.0, API v3.0.0, Component v3.1.3)
              MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v3.1.3)
              MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v3.1.3)
              MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v3.1.3)
              MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v3.1.3)
              MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v3.1.3)
              MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA state: app (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA state: novm (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA state: orted (MCA v2.1.0, API v1.0.0, Component v3.1.3)
               MCA state: tool (MCA v2.1.0, API v1.0.0, Component v3.1.3)
                 MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                MCA coll: self (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: spacc (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
               MCA fcoll: static (MCA v2.1.0, API v2.0.0, Component v3.1.3)
               MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                  MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v3.1.3)
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v3.1.3)
                 MCA pml: v (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                 MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v3.1.3)
                 MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v3.1.3)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v3.1.3)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v3.1.3)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v3.1.3)

谢谢你!

以上是关于如何强制打开mpi 3使用TCP?的主要内容,如果未能解决你的问题,请参考以下文章

如何模拟强制关闭的 TCP 连接?

请问fluent15.0如何打开单机并行计算,Intel MPI

打开 MPI Waitall() 分段错误

Android:强制外部麦克风(插孔输入 3.5mm)保持打开状态

在 FORTRAN / MPI / HDF5 ( 1.10.1 ) 中使用 h5fopen_f 打开文件时出错

如何强制组织模式在另一个框架中打开链接?