并行执行shell进程

Posted 2021-03-27

是否有可用于在Windows批处理文件中并行执行多个进程的工具？我找到了一些有趣的Linux工具（parallel和PPSS），但是，我需要一个适用于Windows平台的工具。

额外奖励：如果该工具还允许在多台机器之间轻松分配流程，并远程运行流程，那就太棒了。

示例：我希望在以下for循环中使用它

PsExec

有限数量的processFile.exe实例并行运行以利用多核CPU。

答案

Linux下的GNU xargs有一个“-P n”开关，可以并行启动“n”进程。

也许cygwin / mingw构建的xargs也支持这个？

然后你可以使用：

for %F in (*.*) do processFile.exe %F

但是，没有花哨的多节点进程产生。

另一答案

编辑 - 我修改了脚本以选择性地显示每个进程的输出

这是一个本机批处理解决方案，可以并行地可靠地运行命令列表，一次不会启动超过n个进程。

它甚至有一个内置的机制，通过PSEXEC将进程分发到特定的CPU或远程机器，但我还没有测试过这个功能。

使这项工作的技巧是通过CMD进程启动每个命令，该进程将stdout或未定义的句柄重定向到锁定文件。该进程将保持对文件的独占锁定，直到它终止。无论进程如何终止（正常退出，崩溃，终止进程），锁定都会立即释放。

主脚本可以通过尝试重定向到同一个锁文件来测试进程是否仍处于活动状态。如果进程仍处于活动状态，则重定向将失败，如果进程已终止，则重定向将成功。

默认情况下，脚本会忽略每个进程的输出。如果以xargs -P 4 processFile < fileList 选项作为第一个参数启动，则它显示每个进程的输出，而不进行交错。

我的演示将进程限制设置为4，并简单地运行一系列不同长度的PING命令。

我在XP，Vista和Windows 7上测试了这个。

/O

这是忽略过程输出的样本运行的输出

@echo off
setlocal enableDelayedExpansion

:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
  set "lockHandle=1"
  set "showOutput=1"
) else (
  set "lockHandle=1^>nul 9"
  set "showOutput="
)

:: The list of commands could come from anywhere such as another file
:: or the output of another command. For this demo I will list the
:: commands within this script - Each command is prefixed with :::
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1

:: Define the maximum number of parallel processes to run.
:: Each process number can optionally be assigned to a particular server
:: and/or cpu via psexec specs (untested).
set "maxProc=4"

:: Optional - Define CPU targets in terms of PSEXEC specs
::           (everything but the command)
::
:: If a CPU is not defined for a proc, then it will be run on the local machine.
:: I haven't tested this feature, but it seems like it should work.
::
:: set cpu1=psexec \server1 ...
:: set cpu2=psexec \server1 ...
:: set cpu3=psexec \server2 ...
:: etc.

:: For this demo force all CPU specs to undefined (local machine)
for /l %%N in (1 1 %maxProc%) do set "cpu%%N="

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
:: Modify the IN () clause as needed to retrieve the list of commands
  set launch=1
  for /f "tokens=* delims=:" %%A in ('findstr /b ":::" "%~f0"') do (
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set cmd!nextProc!=%%A
    if defined showOutput echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting %%A
    2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! %%A
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do 2>nul (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      if defined showOutput echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      if defined showOutput type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    )
  )
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Thats all folks^^!

如果使用显示过程输出的12:24:07.52 - proc1: starting ping /n 05 ::1 12:24:07.52 - proc2: starting ping /n 20 ::1 12:24:07.53 - proc3: starting ping /n 10 ::1 12:24:07.54 - proc4: starting ping /n 15 ::1 12:24:11.60 - proc1: finished ping /n 05 ::1 12:24:11.60 - proc1: starting ping /n 07 ::1 12:24:16.66 - proc3: finished ping /n 10 ::1 12:24:16.66 - proc3: starting ping /n 05 ::1 12:24:17.68 - proc1: finished ping /n 07 ::1 12:24:17.68 - proc1: starting ping /n 20 ::1 12:24:20.72 - proc3: finished ping /n 05 ::1 12:24:20.72 - proc3: starting ping /n 10 ::1 12:24:21.75 - proc4: finished ping /n 15 ::1 12:24:21.75 - proc4: starting ping /n 15 ::1 12:24:26.82 - proc2: finished ping /n 20 ::1 12:24:26.82 - proc2: starting ping /n 07 ::1 12:24:29.86 - proc3: finished ping /n 10 ::1 12:24:32.89 - proc2: finished ping /n 07 ::1 12:24:35.92 - proc4: finished ping /n 15 ::1 12:24:36.93 - proc1: finished ping /n 20 ::1 Thats all folks! 选项运行，则输出如下

/O

另一答案

试试------------------------------------------------------------------------------- 12:24:51.02 - proc1: starting ping /n 05 ::1 ------------------------------------------------------------------------------- 12:24:51.02 - proc2: starting ping /n 20 ::1 ------------------------------------------------------------------------------- 12:24:51.03 - proc3: starting ping /n 10 ::1 ------------------------------------------------------------------------------- 12:24:51.04 - proc4: starting ping /n 15 ::1 =============================================================================== 12:24:55.10 - proc1: finished ping /n 05 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 5, Received = 5, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:24:55.10 - proc1: starting ping /n 07 ::1 =============================================================================== 12:25:00.17 - proc3: finished ping /n 10 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:00.19 - proc3: starting ping /n 05 ::1 =============================================================================== 12:25:01.22 - proc1: finished ping /n 07 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply f