一次生产死锁问题的处理

Posted 2022-09-06 我爱看明朝

一次生产死锁问题的处理

场景

存储制品库maven包，有两张表meta_maven存储groupId、artifactId、version信息，meta_maven_file存储GAV下的具体文件jar、pom、sha1、meta_maven.xml。

场景1: deploy上传snapshot的maven包，用新的meta_maven.xml替换之前的，之前的删除。

场景2: 下载GAV包，检查是否有漏洞：开始更新组件状态为扫描中，从黑鸭子获取扫描结果（超长时间等待），更新组件的扫描结果，有漏洞禁止下载。

场景1伪代码：

@Transactional(propagation=Propagation.REQUIRED, isolation=Isolation.READ_COMMITED, rollbackfor=Exception.class)
public void upload(MetaMavenFile metaMavenFile) 
   MetaMaven metaMaven = metaMavenMapper.select(metaMavenFile);
	
   Result uploadRet = metaMavenFileService.upload(metaMavenFile);	
	
  metaMaven.setModify("xxx");
  metaMavenMapper.update(metaMaven);	


@Transactional(propagation=Propagation.REQUIRED, isolation=Isolation.READ_COMMITED, timeout = 8, rollbackfor=Exception.class)
public Result upload(MetaMavenFile metaMavenFile) 
 MetaMavenFile existMetaMavenFile = metaMavenFileMapper.select(metamavenFile);
 String filePath = fastdfsServer.upload(metaMavenFile);
 existMetaMavenFile.setStorageReference(filePath);
 metaMavenFileMapper.updateByPrimaryKeySelective(existMetaMavenFile);

场景2伪代码：

@Transactional(propagation= Propagation.REQUIRED, isolation= Isolation.READ_COMMITED, timeout = 8, rollbackFor = Exception.class)
public void download(MetaMavenFile metaMavenFile, HttpResponse response)
	mataMaven existMetaMaven = metaMavenMapper.select(metaMavenFile);
mataMavenMapper.updateCompoentStatus(existMetaMaven.getId(), ComponentStatus.SCANNING, null);
	hasSecurityExist(existMetaMaven.getId());


public void hasSecurityExist(Long metaMavenId) 
 	JSONObject scanResult = getblackDuckScanResult(metaMavenId);
 	mataMaven existMetaMaven = metaMavenMapper.select(metaMavenId);
	metaMavenMapper.updateByPrimaryKey(existMetaMavenFile);

事务执行流程

步骤	事务1	事务2
1	begin	begin
2	select * from meta_maven where id = 1
3		select * from meta_maven where id =1
4	update meta_maven set component_status = 1 where id =1
5		update meta_maven set status = 1 where id =1
6	获取黑鸭子扫描结果（超长时间等待）
7		deallock detected
8	update meta_maven set component_status = 2 where id = 1
9	commit	rollback

一般产生死锁的场景（上面的和它不一样，上面的和我们之前介绍的悲观锁发生死锁等待很像）:

步骤	事务1	事务2
1	begin	begin
2	update table1 set col =1 where id =1
3		update table1 set col1 = 2 where id =2
4	update table1 set col1 = 3 where id = 2
5		update table1 set col1 = 4 where id = 1
6	deaded lock	deaded lock

报错信息

 Error updating databse, Cause: org.postgressql.util.PSQLException: ERROR: deadlock detected
 Detail: Process 322869 waits for sharelock in transaction 3523519865;
 blocked by process 369631.
Process 369631 waits foe ShareLock on transaction 3523519865;
blocked by process 369632.
 where: while updating tuple(22892,30) in relation "meta_maven";ERROR: deadlock detected
### SQL: update meta_maven set modifyer = ? where id = ?

怎么处理

我们可以看到，问题很大是出现在hasSecurityExist这个超长等待，

1. 那么我们把这个方法的事务传播机制改为REQUIRED_NEW或者异步执行（这个逻辑支持异步执行的）
2. 大事务拆小，设置手动声明事务的方式来提交

REQUIRED_NEW: 创建一个新的事务，如果当前存在事务，则把当前事务挂起，意思是创建一个新的事务，和原来的事务没有关系

处理思路；

1. 大事务拆小，大事务可能发生死锁，如果业务允许，将大事务拆小
2. 添加合理的索引，没有索引可能会产生表锁，增加死锁发生的概率
3. 降低隔离级别，如果业务允许，将隔离级别调小是比较好的选择，比如将隔离级别从repeat commit 调整为 read commit.

场景2优化伪代码：

@Transactional(propagation= Propagation.REQUIRED, isolation= Isolation.READ_COMMITED, timeout = 8, rollbackFor = Exception.class)
public void download(MetaMavenFile metaMavenFile, HttpResponse response)
	mataMaven existMetaMaven = metaMavenMapper.select(metaMavenFile);
	// 处理死锁问题，这个方法单独抽出来，单独声明事务传播机制

	hasSecurityExist(existMetaMaven.getId());


@Transaction(propagation= Propagation= Propagation.REQUIRES_NEW)
public void updateCompentScaning(Long metaMavenId) 
	mataMavenMapper.updateCompoentStatus(metaMavenId, ComponentStatus.SCANNING, null);


@Transaction(propagation= Propagation= Propagation.REQUIRES_NEW)
public void hasSecurityExist(Long metaMavenId) 
 	JSONObject scanResult = getblackDuckScanResult(metaMavenId);
 	mataMaven existMetaMaven = metaMavenMapper.select(metaMavenId);
	metaMavenMapper.updateByPrimaryKey(existMetaMavenFile);