多线程海量删除查询和批量插入批量的 SQL 死锁
Posted
技术标签:
【中文标题】多线程海量删除查询和批量插入批量的 SQL 死锁【英文标题】：SQL deadlock on multithreaded massive delete query and bulk insert batches 【发布时间】：2011-09-11 15:25:31 【问题描述】：我在 .NET 中有一个批处理，它读取大量文件并将其内容存储到具有以下结构的单个 mysql InnoDB 表中：
`id` int(11) NOT NULL AUTO_INCREMENT,
`Source` varchar(2) NOT NULL,
`Period` char(1) NOT NULL,
`idItem` int(11) NOT NULL,
`StartDate` datetime NOT NULL,
`MoreData` varchar(255)
PRIMARY KEY (`id`),
UNIQUE KEY `Combination` (`Source`,`Period`,`idItem`,`StartDate`),
在哪里
idItem 是外键 Source、Period 和 idItem 是索引 组合是唯一索引。为了执行表更新，我有一个包含两个步骤的事务：
-
删除在我的新数据集的时间间隔内具有 StartDate 的所有旧行。
使用单个 INSERT 语句插入新行（最多 10,000 行）。
开始交易；
从数据中删除 Source = @Source AND Period = @Period AND idItem = @idItem AND StartDate >= @FirstDate AND StartDate
插入数据(..,..,..,..) 值(..,..,..,..)(..,..,..,..)(.., ..,..,..)(..,..,..,..);
提交；
问题在于，当同时从多个线程运行它时（1 个线程正常工作，2 个或更多线程中断），即使每个线程只能更新一组不同的 (Source,Period ,idItem) （没有重叠），我得到一个 1213 异常：Deadlock found when trying to get lock;尝试重新启动。
此问题与此处描述的问题非常相似：SQL deadlock on delete then bulk insert
对于防止这种死锁，您有什么建议？我已经尝试了here 建议的所有方法，但没有令人信服的结果。谢谢！
附件：
------------------------
LATEST DETECTED DEADLOCK
------------------------
110911 12:00:45
*** (1) TRANSACTION:
TRANSACTION 5167, ACTIVE 3 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 4 lock struct(s), heap size 1248, 2 row lock(s), undo log entries 1
MySQL thread id 225, query id 86933 192.168.1.1 root updating
DELETE FROM data WHERE Source = ''mysource'' AND Period = ''D'' AND idItem = 17 AND StartDate >= ''2009-07-22 00:00:00'' AND StartDate <= ''2011-08-29 00:00:00'' ORDER BY StartDate
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 61 page no 389 n bits 344 index `SourcePeriodItemStartDate` of table `crdb`.`data` trx id 5167 lock_mode X waiting
Record lock, heap no 229 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000011; asc ;;
3: len 8; hex 80001245bc591c80; asc E Y ;;
4: len 4; hex 8015b9fd; asc ;;
*** (2) TRANSACTION:
TRANSACTION 5163, ACTIVE 8 sec inserting, thread declared inside InnoDB 198
mysql tables in use 1, locked 1
33 lock struct(s), heap size 6960, 2419 row lock(s), undo log entries 1625
MySQL thread id 224, query id 86924 192.168.1.1 root update
insert data(...)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 61 page no 389 n bits 344 index `SourcePeriodItemStartDate` of table `crdb`.`data` trx id 5163 lock_mode X
Record lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124821527f80; asc H!R ;;
4: len 4; hex 8015c121; asc !;;
Record lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482161c1c0; asc H!a ;;
4: len 4; hex 8015c122; asc ";;
Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124821710400; asc H!q ;;
4: len 4; hex 8015c123; asc #;;
Record lock, heap no 5 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248219ecac0; asc H! ;;
4: len 4; hex 8015c124; asc $;;
Record lock, heap no 6 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124821ae0d00; asc H! ;;
4: len 4; hex 8015c125; asc %;;
Record lock, heap no 7 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124821bd4f40; asc H! O@;;
4: len 4; hex 8015c126; asc &;;
Record lock, heap no 8 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124821cc9180; asc H! ;;
4: len 4; hex 8015c127; asc '';;
Record lock, heap no 9 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124821dbd3c0; asc H! ;;
4: len 4; hex 8015c128; asc (;;
Record lock, heap no 10 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124822099a80; asc H" ;;
4: len 4; hex 8015c129; asc );;
Record lock, heap no 11 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482218dcc0; asc H" ;;
4: len 4; hex 8015c12a; asc *;;
Record lock, heap no 12 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124822281f00; asc H"( ;;
4: len 4; hex 8015c12b; asc +;;
Record lock, heap no 13 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124822376140; asc H"7a@;;
4: len 4; hex 8015c12c; asc ,;;
Record lock, heap no 14 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482246a380; asc H"F ;;
4: len 4; hex 8015c12d; asc -;;
Record lock, heap no 15 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124822746a40; asc H"tj@;;
4: len 4; hex 8015c12e; asc .;;
Record lock, heap no 16 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482283ac80; asc H" ;;
4: len 4; hex 8015c12f; asc /;;
Record lock, heap no 17 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482292eec0; asc H" ;;
4: len 4; hex 8015c130; asc 0;;
Record lock, heap no 18 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124822a23100; asc H" 1 ;;
4: len 4; hex 8015c131; asc 1;;
Record lock, heap no 19 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124822b17340; asc H" s@;;
4: len 4; hex 8015c132; asc 2;;
Record lock, heap no 20 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124826fc1540; asc H& @;;
4: len 4; hex 8015c133; asc 3;;
Record lock, heap no 21 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248270b5780; asc H'' W ;;
4: len 4; hex 8015c134; asc 4;;
Record lock, heap no 22 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248271a99c0; asc H'' ;;
4: len 4; hex 8015c135; asc 5;;
Record lock, heap no 23 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482729dc00; asc H'') ;;
4: len 4; hex 8015c136; asc 6;;
Record lock, heap no 24 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827391e40; asc H''9 @;;
4: len 4; hex 8015c137; asc 7;;
Record lock, heap no 25 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482766e500; asc H''f ;;
4: len 4; hex 8015c138; asc 8;;
Record lock, heap no 26 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827762740; asc H''v''@;;
4: len 4; hex 8015c139; asc 9;;
Record lock, heap no 27 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827856980; asc H'' i ;;
4: len 4; hex 8015c13a; asc :;;
Record lock, heap no 28 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482794abc0; asc H'' ;;
4: len 4; hex 8015c13b; asc ;;;
Record lock, heap no 29 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827a3ee00; asc H'' ;;
4: len 4; hex 8015c13c; asc <;;
Record lock, heap no 30 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827d1b4c0; asc H'' ;;
4: len 4; hex 8015c13d; asc =;;
Record lock, heap no 31 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827e0f700; asc H'' ;;
4: len 4; hex 8015c13e; asc >;;
Record lock, heap no 32 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827f03940; asc H'' 9@;;
4: len 4; hex 8015c13f; asc ?;;
Record lock, heap no 33 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124827ff7b80; asc H'' ;;
4: len 4; hex 8015c140; asc @;;
Record lock, heap no 34 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248280ebdc0; asc H( ;;
4: len 4; hex 8015c141; asc A;;
Record lock, heap no 35 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248283c8480; asc H(< ;;
4: len 4; hex 8015c142; asc B;;
Record lock, heap no 36 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248284bc6c0; asc H(K ;;
4: len 4; hex 8015c143; asc C;;
Record lock, heap no 37 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 80001248285b0900; asc H([ ;;
4: len 4; hex 8015c144; asc D;;
Record lock, heap no 38 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124828798d80; asc H(y ;;
4: len 4; hex 8015c145; asc E;;
Record lock, heap no 39 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124828a75440; asc H( T@;;
4: len 4; hex 8015c146; asc F;;
Record lock, heap no 40 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 8000124828b69680; asc H( ;;
4: len 4; hex 8015c147; asc G;;
Record lock, heap no 41 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482cf1f640; asc H, @;;
4: len 4; hex 8015c148; asc H;;
Record lock, heap no 42 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d013880; asc H- 8 ;;
4: len 4; hex 8015c149; asc I;;
Record lock, heap no 43 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d107ac0; asc H- z ;;
4: len 4; hex 8015c14a; asc J;;
Record lock, heap no 44 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d3e4180; asc H->A ;;
4: len 4; hex 8015c14b; asc K;;
Record lock, heap no 45 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d4d83c0; asc H-M ;;
4: len 4; hex 8015c14c; asc L;;
Record lock, heap no 46 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d5cc600; asc H-\ ;;
4: len 4; hex 8015c14d; asc M;;
Record lock, heap no 47 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d6c0840; asc H-l @;;
4: len 4; hex 8015c14e; asc N;;
Record lock, heap no 48 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482d7b4a80; asc H-J ;;
4: len 4; hex 8015c14f; asc O;;
Record lock, heap no 49 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482da91140; asc H- @;;
4: len 4; hex 8015c150; asc P;;
Record lock, heap no 50 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482db85380; asc H- S ;;
4: len 4; hex 8015c151; asc Q;;
Record lock, heap no 51 PHYSICAL RECORD: n_fields 5; compact format; info bits 32
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000010; asc ;;
3: len 8; hex 800012482dc795c0; asc H- ;;
4: len 4; hex 8015c152; asc R;;
(around 250 more record locks like this)
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 61 page no 389 n bits 544 index `SourcePeriodItemStartDate` of table `crdb`.`data` trx id 5163 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 229 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 2; hex 4242; asc mysource;;
1: len 1; hex 44; asc D;;
2: len 4; hex 80000011; asc ;;
3: len 8; hex 80001245bc591c80; asc E Y ;;
4: len 4; hex 8015b9fd; asc ;;
*** WE ROLL BACK TRANSACTION (1)
【问题讨论】：
这个问题你解决了吗？我对 oracle 也有同样的看法。 我不认为我以任何对其他人有用的方式解决了这个问题（虽然这是很久以前的事了）。 【参考方案1】：将ORDER BY StartDate 添加到删除中。 （因此行会按特定顺序锁定）
【讨论】：
谢谢，但它似乎没有帮助:(，我不断收到异常（只有 2 个线程）。 为什么不把插入放到另一个事务中呢？ 拆分成两个事务惨遭失败，出现另一个异常（8线程）：“连接尝试失败，因为连接方在一段时间后没有正确响应，或者建立连接失败，因为连接的主机有没有回应”。 此外，在同一事务中进行删除和插入的目的是拦截对同一（Source、Period、idItem）内行的冲突更新，以防另一个进程（不是线程）尝试访问数据库并进行其他类型的修改。否则，一旦删除了行，就可能无法再插入新的行，因为另一个进程已经插入了其他行。理想情况下，我想在同一个事务中包含两个操作以及另一个写入另一个表以指定数据集特征的操作。 如果你已经挂钩了数据库，听起来拆分成两个事务的效果非常好。如果有其他进程正在运行，您可能需要节流，以使您的占用率低于 100%。所有线程的目的是什么？并行的四个事务不会比 4 个串行事务更快地完成。尝试使用 1、2、3 和 4 个线程并测量完成时间。您可能会发现 2 或 3 是最佳选择。以上是关于多线程海量删除查询和批量插入批量的 SQL 死锁的主要内容，如果未能解决你的问题，请参考以下文章