MySql批量插入优化Sql执行效率

Posted 真爱无限

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了MySql批量插入优化Sql执行效率相关的知识,希望对你有一定的参考价值。

itemcontractprice数量1万左右,每条itemcontractprice 插入5条日志。


                                updateInsertSql.AppendFormat("UPDATE itemcontractprice AS p INNER JOIN foreigncurrency AS f ON p.ForeignCurrencyId = f.ContractPriceId SET p.RemainPrice = f.RemainPrice * 0,p.BuyOutPrice = f.BuyOutPrice * 0,p.ReservedPrice = f.ReservedPrice * 0,p.CollectedPrice = f.CollectedPrice * 0,p.AccessPrice = f.AccessPrice * 0 WHERE p.CurrencyId = 1 AND p.date BETWEEN '2:yyyy-MM-dd' AND '3:yyyy-MM-dd';", rate.ExchangeRate, exchangeRate.CurrencyId, rate.BeginDate, rate.EndDate);

                                updateInsertSql.AppendFormat("INSERT INTO `itemcontractpricelog`(`ContractPriceType`,`ContractPrice`,`FcContractPrice`,`IsExpire`,`LogRemark`,`CreatedByName`,`CreatedById`,`CreatedDate`,`LogTypeId`,`ProviderId`,`StageId`,`Date`,`CurrencyId`,`ContractPriceId`,`StockPattern`,`ItemId`) SELECT 0,c.RemainPrice,f.RemainPrice,c.RemainIsExpire,'外币汇率调整,重新计算人民币底价','job',0,NOW(),5,c.ProviderId,c.StageId,c.Date,c.CurrencyId,c.ContractPriceId,0,c.ItemId  FROM itemcontractprice AS c INNER JOIN foreigncurrency AS f ON c.ForeignCurrencyId = f.ContractPriceId WHERE c.CurrencyId=0 AND c.date BETWEEN '1:yyyy-MM-dd' AND '2:yyyy-MM-dd';", exchangeRate.CurrencyId, rate.BeginDate, rate.EndDate);

                                updateInsertSql.AppendFormat(" INSERT INTO `itemcontractpricelog`(`ContractPriceType`,`ContractPrice`,`FcContractPrice`,`IsExpire`,`LogRemark`,`CreatedByName`,`CreatedById`,`CreatedDate`,`LogTypeId`,`ProviderId`,`StageId`,`Date`,`CurrencyId`,`ContractPriceId`,`StockPattern`,`ItemId`) SELECT 1,c.BuyOutPrice,f.BuyOutPrice,c.BuyOutIsExpire,'外币汇率调整,重新计算人民币底价','job',0,NOW(),5,c.ProviderId,c.StageId,c.Date,c.CurrencyId,c.ContractPriceId,0,c.ItemId  FROM itemcontractprice AS c INNER JOIN foreigncurrency AS f ON c.ForeignCurrencyId = f.ContractPriceId WHERE c.CurrencyId=0 AND c.date BETWEEN '1:yyyy-MM-dd' AND '2:yyyy-MM-dd';", exchangeRate.CurrencyId, rate.BeginDate, rate.EndDate);

                                updateInsertSql.AppendFormat("INSERT INTO `itemcontractpricelog`(`ContractPriceType`,`ContractPrice`,`FcContractPrice`,`IsExpire`,`LogRemark`,`CreatedByName`,`CreatedById`,`CreatedDate`,`LogTypeId`,`ProviderId`,`StageId`,`Date`,`CurrencyId`,`ContractPriceId`,`StockPattern`,`ItemId`) SELECT 2,c.ReservedPrice,f.ReservedPrice,c.ReservedIsExpire,'外币汇率调整,重新计算人民币底价','job',0,NOW(),5,c.ProviderId,c.StageId,c.Date,c.CurrencyId,c.ContractPriceId,0,c.ItemId  FROM itemcontractprice AS c INNER JOIN foreigncurrency AS f ON c.ForeignCurrencyId = f.ContractPriceId WHERE c.CurrencyId=0 AND c.date BETWEEN '1:yyyy-MM-dd' AND '2:yyyy-MM-dd';", exchangeRate.CurrencyId, rate.BeginDate, rate.EndDate);

                                updateInsertSql.AppendFormat("INSERT INTO `itemcontractpricelog`(`ContractPriceType`,`ContractPrice`,`FcContractPrice`,`IsExpire`,`LogRemark`,`CreatedByName`,`CreatedById`,`CreatedDate`,`LogTypeId`,`ProviderId`,`StageId`,`Date`,`CurrencyId`,`ContractPriceId`,`StockPattern`,`ItemId`) SELECT 3,c.CollectedPrice,f.CollectedPrice,c.CollectedIsExpire,'外币汇率调整,重新计算人民币底价','job',0,NOW(),5,c.ProviderId,c.StageId,c.Date,c.CurrencyId,c.ContractPriceId,0,c.ItemId  FROM itemcontractprice AS c INNER JOIN foreigncurrency AS f ON c.ForeignCurrencyId = f.ContractPriceId WHERE c.CurrencyId=0 AND c.date BETWEEN '1:yyyy-MM-dd' AND '2:yyyy-MM-dd';", exchangeRate.CurrencyId, rate.BeginDate, rate.EndDate);
                                updateInsertSql.AppendFormat("INSERT INTO `itemcontractpricelog`(`ContractPriceType`,`ContractPrice`,`FcContractPrice`,`IsExpire`,`LogRemark`,`CreatedByName`,`CreatedById`,`CreatedDate`,`LogTypeId`,`ProviderId`,`StageId`,`Date`,`CurrencyId`,`ContractPriceId`,`StockPattern`,`ItemId`) SELECT 4,c.AccessPrice,f.AccessPrice,c.AccessIsExpire,'外币汇率调整,重新计算人民币底价','job',0,NOW(),5,c.ProviderId,c.StageId,c.Date,c.CurrencyId,c.ContractPriceId,0,c.ItemId  FROM itemcontractprice AS c INNER JOIN foreigncurrency AS f ON c.ForeignCurrencyId = f.ContractPriceId WHERE c.CurrencyId=0 AND c.date BETWEEN '1:yyyy-MM-dd' AND '2:yyyy-MM-dd';", exchangeRate.CurrencyId, rate.BeginDate, rate.EndDate);
                                //var curContractPriceList = itemContractPriceList.Where(o => o.CurrencyId == exchangeRate.CurrencyId && o.Date >= rate.BeginDate && o.Date <= rate.EndDate).ToList();
                                logger.InfoFormat("底价更新和日志sql:0", updateInsertSql.ToString());
                                //if (curContractPriceList.Count == 0) continue;
                                int effctRows = 0;
                                using (var tran = UnitOfWorkManager.Begin())
                                
                                    effctRows = taskRepository.ExecuteSql(updateInsertSql.ToString(), false);
                                    tran.Complete();
                                
                                logger.InfoFormat("底价更新影响行数:0", effctRows);

正常情况下大概20秒钟就ok.

之前是用EF操作,查询出来 ,要耗时,然后再组装 update语句 ,然后再插入日志(每条数据5条日志),这个网络交互的时间加上数据库连接打开关闭的时间,总的执行时间,大概10多分钟。

用sql语句批量操作,可以说效率提升了 40倍,就是大量数据的传输和数据库的处理次数耗时。


所以说,软件开发不是开发完成就行,而是要解决性能上的问题,这才是开发的进阶。








以上是关于MySql批量插入优化Sql执行效率的主要内容,如果未能解决你的问题,请参考以下文章

MySQL批量SQL插入性能优化

MySQL批量SQL插入性能优化详解

MySQL批量SQL插入各种性能优化

MySQL批量SQL插入性能优化

批量向MySQL导入1000万条数据的优化

mysql 大量数据插入优化