表观死锁为未分配的挂起任务创建紧急线程
Posted
技术标签:
【中文标题】表观死锁为未分配的挂起任务创建紧急线程【英文标题】：APPARENT DEADLOCK Creating emergency threads for unassigned pending tasks 【发布时间】：2013-01-21 17:29:01 【问题描述】：我正在使用带有 mybatis 的 mysql，我在我们的实时服务器上遇到了这个错误
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6538f8f2
-- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
我不明白为什么会出现这个错误是因为我的 C3P0 设置？我的C3P0设置是这样的
----开始更新-----
下面是我的 spring-servlet.xml 配置
我更新了数据源bean为
<bean id="datasource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close" p:driverClass="com.mysql.jdbc.Driver"
p:jdbcUrl="jdbc:mysql://localhost/jdb" p:user="root" p:password="root"
p:acquireIncrement="10"
p:idleConnectionTestPeriod="60"
p:maxPoolSize="100"
p:maxStatements="0"
p:minPoolSize="10"
p:initialPoolSize="10"
p:statementCacheNumDeferredCloseThreads="1" />
<!-- Declare a transaction manager -->
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager"
p:dataSource-ref="datasource" />
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="datasource" />
</bean>
<!-- scan for mappers and will automatically scan the whole classpath for xmls -->
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
<property name="sqlSessionFactory" ref="sqlSessionFactory" />
<property name="basePackage" value="com.mycom.myproject.db.mybatis.dao" />
</bean>
在我的 Dao 类中，我调用映射器方法，例如
myDao.updateRecords()
这是我的服务类方法
@Override
public List<UserDetailedBean> selectAllUsersDetail(long groupId, List<Long> ids)
List<UserDetailedBean> usersDetailList = null;
try
usersDetailList = userDao.selectAllUsersDetail(groupId, ids);
catch (Exception e)
e.printStackTrace();
return usersDetailList;
在 Dao 类中，我只注入映射器。
@Resource
private UserMapper userMapper;
@Override
public List<UserDetailedBean> selectAllUsersDetail(long groupId, List<Long> ids)
return userMapper.selectAllUsersDetail(groupId,ids);
---结束更新-----
如果需要任何其他信息，请告诉我。
这是完整的堆栈跟踪
[ WARN] 2013-01-08 20:13:39 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@70497e11 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
[ WARN] 2013-01-08 20:13:39 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@70497e11 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2e81b8c5 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4689a55d (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@76c7a0d8 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2c1101d4
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@108f1be6
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2370a188
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@377cf9e5
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6dfa45d8
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@49ffa050
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2d760a24
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2549)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2618)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1557)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4984)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:51)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4962)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2549)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2618)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1557)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4984)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:51)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4962)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2549)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2618)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1557)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4984)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:51)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4962)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
---更新----
当我将 p:statementCacheNumDeferredCloseThreads="1" 添加到 datasouce bean 时，我收到以下错误
Error creating bean with name 'sqlSessionFactory' defined in ServletContext resource [/WEB-INF/spring-servlet.xml]:
Cannot resolve reference to bean 'datasource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'datasource' defined in ServletContext resource [/WEB-INF/spring-servlet.xml]:
Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'statementCacheNumDeferredCloseThreads' of bean class [com.mchange.v2.c3p0.ComboPooledDataSource]:
Bean property 'statementCacheNumDeferredCloseThreads' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
【问题讨论】：
请举例说明 updateRecords() dao 方法的实现。 嗨，Taky，请更新我的问题。请检查并让我知道是否需要其他任何内容。谢谢 另外请说明如何在 DAO 中创建映射器。您的应用程序中有多少个映射器？ 嗨，Taky，我更新了我的问题。我只是将映射器注入到我的 Dao 类中，我的应用程序中有 20 个映射器。谢谢 在 c3p0 0.9.1.x 版本中似乎没有解决这个问题。在 maven repo 的最后一个版本中，只有 pre：mvnrepository.com/artifact/com.mchange/c3p0/0.9.2-pre8。因此，您应该尝试使用您的特定应用程序。尝试将 maxStatement 设置为 0 或一些较大的值，例如 - 200 或 300。 【参考方案1】：请查看后续步骤以解决问题：
在 ComboPooledDataSource 中增加 p:maxStatements。
将 p:maxStatements 设置为 0。例如，在 Firebird 中，此 hack 可以使用 ComboPooledDataSource。
确保在应用程序中关闭 SqlSession。更加注意集中执行数据库操作。在我的 mySql JDBC 驱动程序版本中：当对象被垃圾收集时，mysql-connector-java 5.1.8 连接会自动关闭。因此，如果您不大量使用数据库，则在您的情况下连接不应该泄漏。尽管如此，您必须确保您关闭了 myBatis SqlSession，它包装了与 DB 的 jdbc 连接。
同样JDBC3 Connection and Statement Pooling，您可以尝试在 c3p0 配置中将 statementCacheNumDeferredCloseThreads 设置为 1。
【讨论】：
嗨，Taky，感谢您的回复。我正在使用带有spring的mybatis并使用sqlSessionFactory而不是sqlSession。 sqlSessionFactory 不会自动关闭连接吗？ 如何使用SqlSessionFactory？请显示使用示例。因此，MyBatis Doc (mybatis.org/core/getting-started.html) 你必须创建 SqlSession 来执行对 DB 的查询。 嗨，Taky，请检查我如何使用 sqlSessionFactory 的更新问题 如果应用服务器中的数据库凭据不匹配也会出现此问题【参考方案2】：来自http://www.mchange.com/projects/c3p0/#other_ds_configuration
numHelperThreads 和 maxAdministrativeTaskTime 有助于配置 DataSource 线程池的行为。默认情况下，每个 DataSource 只有三个关联的帮助线程。如果性能似乎在重负载下拖累，或者如果您通过 JMX 或直接检查 PooledDataSource 观察到“待处理任务”的数量通常大于零，请尝试增加 numHelperThreads。 maxAdministrativeTaskTime 对于遇到无限期挂起的任务和“APPARENT DEADLOCK”消息的用户可能很有用。 （更多信息请参见附录 A。）
maxAdministrativeTaskTime 默认值：0 在 c3p0 的线程池将尝试中断一个明显挂起的任务之前的几秒钟。很少有用。 c3p0 的许多功能不是由客户端线程执行的，而是由内部线程池异步执行的。 c3p0 的异步性直接增强了客户端性能，并通过确保在非锁持有线程中执行慢速 jdbc 操作来最小化持有关键锁的时间长度。但是，如果这些任务中的一些“挂起”，即它们在很长一段时间内既没有成功也没有失败并出现异常，c3p0 的线程池可能会耗尽并且管理任务会备份。如果任务只是很慢，解决问题的最佳方法是通过 numHelperThreads 增加线程数。但是如果任务有时会无限期挂起，您可以使用此参数在任务超过设定的时间限制时强制调用任务线程的 interrupt() 方法。 [c3p0 最终将通过发出“明显的死锁”信号（您将在日志中将其视为警告）、替换线程池任务线程和中断（）原始线程来从挂起的任务中恢复。但是让池进入表观死锁然后恢复意味着在某些时期内，c3p0 的性能会受到损害。因此，如果您看到这些消息，增加 numHelperThreads 并设置 maxAdministrativeTaskTime 可能会有所帮助。 maxAdministrativeTaskTime 应该足够大，以使任何从数据库获取连接、测试连接或两次破坏连接的合理尝试都可以在设定的时间内成功或失败。零（默认值）意味着任务永远不会中断，这是大多数情况下最好和最安全的策略。如果任务很慢，请分配更多线程。如果任务永远挂起，请尝试找出原因，同时设置 maxAdministrativeTaskTime 可能会有所帮助。
The default is 3 for numHelperThreads , increase this to 8-10
setting maxAdministrativeTaskTime will help
【讨论】：
【参考方案3】：在我的情况下，应用程序的内存太少是原因。使用的数据库是 H2 或 SQLite（都在这个应用程序中使用）。
第一个症状是上面报告的这些 WARN 日志行：
12006925 [C3P0PooledConnectionPoolManager[identityToken->2rvy8f9szmpczp1k2dm1g|33af2d37]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6d3a9c65 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
12016284 [C3P0PooledConnectionPoolManager[identityToken->2rvy8f9szmpczp1k2dm1g|3d98d1b]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@44565f94 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
12051847 [C3P0PooledConnectionPoolManager[identityToken->2rvy8f9szmpczp1k2dm1g|5703a6aa]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@c9f37e2 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
12085128 [C3P0PooledConnectionPoolManager[identityToken->2rvy8f9szmpczp1k2dm1g|4e50d42b]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6f1927b7 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
12085128 [C3P0PooledConnectionPoolManager[identityToken->2rvy8f9szmpczp1k2dm1g|78fa7f84]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@22c22b50 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
12172644 [C3P0PooledConnectionPoolManager[identityToken->2rvy8f9szmpczp1k2dm1g|e8e88fa]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@745a644f -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
很长一段时间后出现异常，包括揭示：
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
问题是可重现的。为应用程序提供更多内存 (-Xmx8G) 修复了它。
【讨论】：
以上是关于表观死锁为未分配的挂起任务创建紧急线程的主要内容，如果未能解决你的问题，请参考以下文章
Kotlin 协程协程的挂起和恢复 ② ( 协程挂起 和 线程阻塞 对比 )