分布式任务执行之mpirun

Posted wx62b90c63c9df5

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了分布式任务执行之mpirun相关的知识,希望对你有一定的参考价值。

假如你有很多VM, 你想快速利用VM来完成一些任务, 比如并行执行一些命令。网上有很多解决方案, 或者自己也可以写一些工具来完成, 但是有一个工具非常强大,而且是开源的。 下面简单介绍下mpirun.

installation

yum -y install openmpi-devel
# add path to ~/.bashrc for all nodes
export PATH=$PATH:/usr/lib64/openmpi/bin

同一个vm执行多个进程

mpirun --allow-run-as-root -np 3 --display-map echo rock
Data for JOB [2092,1] offset 0

======================== JOB MAP ========================

Data for node: drmcyc-fur5-1-01 Num slots: 12 Max slots: 0 Num procs: 3
Process OMPI jobid: [2092,1] App: 0 Process rank: 0
Process OMPI jobid: [2092,1] App: 0 Process rank: 1
Process OMPI jobid: [2092,1] App: 0 Process rank: 2

=============================================================
rock
rock
rock

不同vm执行单个进程

mpirun --allow-run-as-root -np 2 --display-map -host drmcyc-fur5-1-00 --host drmcyc-fur5-1-01 echo rock
Data for JOB [32409,1] offset 0

======================== JOB MAP ========================

Data for node: drmcyc-fur5-1-00 Num slots: 1 Max slots: 0 Num procs: 1
Process OMPI jobid: [32409,1] App: 0 Process rank: 0

Data for node: drmcyc-fur5-1-01 Num slots: 1 Max slots: 0 Num procs: 1
Process OMPI jobid: [32409,1] App: 0 Process rank: 1

=============================================================
rock
rock

不同vm执行多个进程

mpirun --allow-run-as-root -np 4 --display-map -host drmcyc-fur5-1-00,drmcyc-fur5-1-00,drmcyc-fur5-1-01,drmcyc-fur5-1-01 echo rock
Data for JOB [31300,1] offset 0

======================== JOB MAP ========================

Data for node: drmcyc-fur5-1-00 Num slots: 2 Max slots: 0 Num procs: 2
Process OMPI jobid: [31300,1] App: 0 Process rank: 0
Process OMPI jobid: [31300,1] App: 0 Process rank: 1

Data for node: drmcyc-fur5-1-01 Num slots: 2 Max slots: 0 Num procs: 2
Process OMPI jobid: [31300,1] App: 0 Process rank: 2
Process OMPI jobid: [31300,1] App: 0 Process rank: 3

=============================================================
rock
rock
rock
rock

host配置文件

cat tmp/hosts 
10.207.65.19 slots=2 max_slots=20
10.207.69.19 slots=3 max_slots=20

mpirun --allow-run-as-root -np 4 --display-map --hostfile tmp/hosts echo rock
Data for JOB [25634,1] offset 0

======================== JOB MAP ========================

Data for node: 10.207.65.19 Num slots: 2 Max slots: 20 Num procs: 2
Process OMPI jobid: [25634,1] App: 0 Process rank: 0
Process OMPI jobid: [25634,1] App: 0 Process rank: 1

Data for node: drmcyc-fur5-1-01 Num slots: 3 Max slots: 20 Num procs: 2
Process OMPI jobid: [25634,1] App: 0 Process rank: 2
Process OMPI jobid: [25634,1] App: 0 Process rank: 3

=============================================================
rock
rock
rock
rock


是不是很好用?

以上是关于分布式任务执行之mpirun的主要内容,如果未能解决你的问题,请参考以下文章

mpirun 找不到指定的可执行文件,用 java

使用 mpirun 执行我的程序会大大降低性能

mpirun 的自定义中断处理程序

赫拉(hera)分布式任务调度系统之操作文档

Flink1.6系列之—分布式运行环境

基础概念 之 Hadoop family