如何在 Python 中通过 apply_async() 传递 gurobipy.Model 变量？

Posted 2023-02-16

【中文标题】如何在 Python 中通过 apply_async() 传递 gurobipy.Model 变量？

【英文标题】：How to pass gurobipy.Model variables through apply_async() in Python?

【发布时间】：2021-01-11 11:41:12

【问题描述】：

总结 我是 python 并行计算的新手。我使用 Gurobi 建立了一个 DEA 模型并计算了每个 DMU（决策单元）的效率。为了减少总的计算时间，我将模型分为两步求解：

Step1，定义k个模型 Step2，并行优化 k 个模型。

Step1 正确且可执行。但是在第二步中，当我通过multiprocessing.Pool.apply_async() 将对象参数“gurobipy.Model”传递给我定义的函数Solve() 时，出现了TypeError: can't pickle PyCapsule objects。并且函数Solve() 没有被执行。如何使用apply_async函数传递gurobipy.Model变量，或者有没有其他并行的方法传递gurobipy.Model变量？

详情 以下是主程序

from multiprocessing import Pool
import multiprocessing
from gurobipy import *
import gurobipy as gp
import numpy as np
import pandas as pd
import time

def runComputationgurobi(Root, FileName, ResultFileName, numInput, numOutput):
    '''
    input：root path, file name, number of input unit, number of output unit
    output：Excel file (including DMU number, best solution(efficiency), modeling time, solving time)
    '''
    #Data peprocessing
    df = pd.read_csv(f"Root/FileName", header=None)   #download data
    D = np.array(df)                                      #convert to ndarray
    transD = D.transpose()                                #transpose ndarray
    outputs = []                                          #empty list to store best solutions
    
    scale, S = transD.shape                               #scale : numInput+numOutput；S : total number of DMUs

    print("Build k models...")
    #Step1: Modeling
    '''
    call BuildGurobiModel(list of download data, number of input unit,number of output unit)
    return：k modeling times(list[float])、k Gurobi models(list[gurobipy.Model])
    '''
    build_time_house, model_house = BuildGurobiModels(transD, numInput, numOutput)

    print("Parallel computing k models...")
    #Step2: Parallel optimization model
    '''
    call Solve(kth Gurobi model)
    return：k best solutions(efficiency)(float)、k solving times(float)
    '''
    temp = []
    pool = multiprocessing.Pool(4)
    print("Start parallel solve")
    start_time = time.time()
    for k in range(S):
        temp.append([k+1, build_time_house[k], pool.apply_async(Solve, args=(model_house[k], ))])
    pool.close()
    pool.join()
    print(f"time.time() - start_times")

    for k, build_time, _return in temp:
        outputs.append([k, _return.get()[0], build_time, _return.get()[1]])  #_return.get()=(obj_efficiency, solve_time, )
    
    #Output Excel
    pd.DataFrame(np.array(outputs)).to_excel(f"Root/result_parallel_matrix_ChgRHS.xlsx", header=["DMU", "obj_efficiency", "build_time", "solve_time"], index=False)

if __name__=="__main__":
    rootPath = "C:/Users/MB516/Documents/source/Python Scripts/Parallel_processing"
    file_name = "test.csv"
    resultfile_name = "result.csv"

    numInput = 2
    numOutput = 3

    start_time = time.time()
    runComputationgurobi(rootPath, file_name, resultfile_name, numInput, numOutput)
    parallel_solveTime = time.time() - start_time

    print(f"solveTime:parallel_solveTime")

构建 k 个模型：

def BuildGurobiModels(transD, numInput, numOutput):
    '''
    input: list of download data(list), number of input unit(int),number of output unit(int)
    return: k modeling times(list[float]), k Gurobi models(list[gurobipy.Model])
    '''
    #Data peprocessing
    model_house = []
    build_time_house = []
    scale, S = transD.shape  #scale : numInput+numOutput；S : total number of DMUs

    for k in range(S):
        #Define model
        start_time = time.time()
        model = gp.Model(f"NaiveDEAk+1")
        model.setParam("OutputFlag", 0) # 0: disables solver output
        model.setParam("Method", 0)     # 0: primal simplex

        #Define variables
        #define lambda
        lambdaarray = model.addVars(S, lb = 0.0, ub = GRB.INFINITY, vtype = GRB.CONTINUOUS)

        #define theta
        theta = model.addVar(lb = -GRB.INFINITY, ub = GRB.INFINITY, vtype=GRB.CONTINUOUS, name="theta")
        model.update()

        #Set the objective
        model.setObjective(theta, GRB.MINIMIZE)

        #Define constraints
        #input constraint
        model.addConstrs((LinExpr(transD[i], lambdaarray.values()) <=transD[i, k]*theta for i in range(numInput)), name = "Input")
        model.update()

        #output constraint
        model.addConstrs((LinExpr(transD[j], lambdaarray.values()) >=transD[j, k] for j in range(numInput, scale)), name = "Output")
        model.update()

        #convexity constraint
        model.addConstr(quicksum(lambdaarray)==1, name="Convexity")
        model.update()

        build_time = time.time() - start_time   #modeling time

        model_house.append([model])
        build_time_house.append([build_time])

    return build_time_house, model_house

求解第 k 个模型：

def Solve(model):
    '''
    input: kth Gurobi model(gurobipy.Model)
    return：k best solutions(efficiency)(float), k solving times(float)
    ''' 
    print("Start Solve!!!!!!")      
    #Solve
    start_time = time.time()
    model.optimize()
    solve_time = time.time() - start_time

    #print
    objvalue = model.getObjective()
    getobjv = objvalue.getValue()

当我运行代码时，结果如下所示。

Build k models...
Parallel computing k models...
0.53267502784729s
Traceback (most recent call last):
  File "c:/Users/MB516/Documents/source/Python Scripts/Parallel_processing/ENGLIFH_TEST_PARALLEL.py", line 124, in <module>
    runComputationgurobi(rootPath, file_name, resultfile_name, numInput, numOutput)
  File "c:/Users/MB516/Documents/source/Python Scripts/Parallel_processing/ENGLIFH_TEST_PARALLEL.py", line 47, in runComputationgurobi
    outputs.append([k, _return.get()[0], build_time, _return.get()[1]])  #_return.get()=(obj_efficiency, solve_time, )
TypeError: can't pickle PyCapsule objects

它没有执行步骤 2 的 Solve 函数，因为它没有打印出“Start Solve!!!!!!”在函数Solve() 中。以及下面的程序

for k, build_time, _return in temp:
        outputs.append([k, _return.get()[0], build_time, _return.get()[1]]) #_return.get()=(obj_efficiency, solve_time, )

有TypeError: can't pickle PyCapsule objects。我怎么解决这个问题 ？提前感谢您的回答！

环境

操作系统：Windows 10（64 位） 编译器：Visual Studio Code 1.49.1 Python：3.6.10 Gurobi（求解器）：9.0.3

【问题讨论】：

请尝试缩小问题范围。似乎涉及多种工具，目前尚不清楚错误源自何处。

嗨，@mattmilten，我试图缩小摘要中的问题范围。问题是“如何使用apply_async 传递gurobipy.Model 对象”或“哪种并行操作方法可以输入gurobipy.Model 对象”。另外，我猜测TypeError: can't pickle PyCapsule objects出现是因为我自己定义的函数Solve()没有执行。

【参考方案1】：

这是您可以在 Python 中并行创建和求解多个模型的方法：

import multiprocessing as mp
import gurobipy as gp

def solve_model(input_data):
    with gp.Env() as env, gp.Model(env=env) as model:
        # define model
        model.optimize()
        # retrieve data from model

if __name__ == '__main__':
    with mp.Pool() as pool:
        pool.map(solve_model, [input_data1, input_data2, input_data3]

更多信息请参考full guide。

【讨论】：

感谢您的回答。我会试试这个方法。另外，这样可以只建一次模型，多次通过吗？根据我的DEA模型，我想要的是“只构建一次模型”，然后只需要修改模型的某个约束的RHS，并做k次（k为正整数）。

不，您需要创建模型的多个副本。一个完全不同的方法可能是multi-scenario optimization。

你好，@mattmilten，经过练习，我又回到了原来的问题。让我用link的例子来说明。在示例中，当定义了基本模型时，如何并行修改每个场景（示例没有并行修改每个场景）？要并行修改它们，我必须使用pool.apply_async 等函数并传递gurobipy.Model 变量。但这又回到了“pool.apply_async 不支持传递 gurobipy.Model 变量”的问题。

并行运算函数是否支持gurobipy.Model变量的传递？

多场景解决方案（就像正常的优化一样）在内部使用并行化，因此您应该检查是否甚至可以使用（手动）外部并行化来击败它。