BoneCP 抛出“SQLException:连接已关闭!”批量插入 MySQL 时
Posted
技术标签:
【中文标题】BoneCP 抛出“SQLException:连接已关闭!”批量插入 MySQL 时【英文标题】:BoneCP throws "SQLException: Connection is closed!" when batch inserting into MySQL 【发布时间】:2014-05-04 20:01:12 【问题描述】:我的任务是使用 BoneCP 与 jOOQ 和 Spring 建立一个项目,但我在这样做时遇到了一些困难。对我的 mysql 数据库进行单独插入工作非常好,但是使用 190 000 个对象这样做需要将近 20 分钟,所以为了加快速度,我想一次使用 100 个批量插入。但是,这会引发以下异常:
org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Connection is closed!
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:288)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:849)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:826)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:496)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:266)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at com.theshahin.service.YmsLinkDataService$$EnhancerBySpringCGLIB$$b9b6e447.create(<generated>)
at com.theshahin.integration.YmsLinkDataServiceTest.foo(YmsLinkDataServiceTest.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:232)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:175)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
Caused by: java.sql.SQLException: Connection is closed!
at com.jolbox.bonecp.ConnectionHandle.checkClosed(ConnectionHandle.java:459)
at com.jolbox.bonecp.ConnectionHandle.rollback(ConnectionHandle.java:1270)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:285)
... 44 more
(值得一提的是,这个异常是在第一次批量查询时抛出的,因此在此之前没有执行任何查询)。这是我的 applicationContext.xml,它基于 jOOQ 教程中的一个(你可以在这里找到它:http://www.jooq.org/doc/3.3/manual/getting-started/tutorials/jooq-with-spring/):
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.2.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.2.xsd
http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-3.2.xsd">
<context:component-scan base-package="com.theshahin" />
<context:property-placeholder location="classpath:application.properties" ignore-resource-not-found="false"/>
<tx:annotation-driven transaction-manager="transactionManager"/>
<bean id="dataSource" class="com.jolbox.bonecp.BoneCPDataSource" destroy-method="close">
<property name="driverClass" value="$db.driver"/>
<property name="jdbcUrl" value="$db.url"/>
<property name="username" value="$db.username"/>
<property name="password" value="$db.password"/>
<property name="idleConnectionTestPeriod" value="60"/>
<property name="idleMaxAge" value="240"/>
<property name="maxConnectionsPerPartition" value="30"/>
<property name="minConnectionsPerPartition" value="10"/>
<property name="partitionCount" value="3"/>
<property name="acquireIncrement" value="5"/>
<property name="statementsCacheSize" value="100"/>
<property name="releaseHelperThreads" value="3"/>
</bean>
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="transactionAwareDataSource"
class="org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy">
<constructor-arg ref="dataSource" />
</bean>
<bean class="org.jooq.impl.DataSourceConnectionProvider" name="connectionProvider">
<constructor-arg ref="transactionAwareDataSource" />
</bean>
<bean id="dsl" class="org.jooq.impl.DefaultDSLContext">
<constructor-arg ref="config" />
</bean>
<bean id="jooqToSpringExceptionTransformer" class="com.theshahin.exception.JOOQToSpringExceptionTransformer"/>
<bean class="org.jooq.impl.DefaultConfiguration" name="config">
<constructor-arg index="0" ref="connectionProvider" />
<constructor-arg index="1"><null /></constructor-arg>
<constructor-arg index="2"><null /></constructor-arg>
<constructor-arg index="3">
<list>
<bean class="org.jooq.impl.DefaultExecuteListenerProvider">
<constructor-arg index="0" ref="jooqToSpringExceptionTransformer"/>
</bean>
</list>
</constructor-arg>
<constructor-arg index="4"><null /></constructor-arg>
<constructor-arg index="5"><value type="org.jooq.SQLDialect">$jooq.sql.dialect</value></constructor-arg>
<constructor-arg index="6"><null /></constructor-arg>
<constructor-arg index="7"><null /></constructor-arg>
</bean>
这是用于将记录保存到 MySQL 数据库的代码。 (注意:注释掉的代码是我用于单个插入的代码)
@Service
public class YmsLinkDataService extends BaseService
@Transactional
public void create(List<YmsLinkDataRecord> records)
dsl.batchInsert(records).execute();
// dsl.insertInto(YMS_LINK_DATA, YMS_LINK_DATA.SITE_ID,
// YMS_LINK_DATA.SITE_TYPE, YMS_LINK_DATA.TIME, YMS_LINK_DATA.URL,
// YMS_LINK_DATA.KEYWORD).values(linkData.getSiteId(),
// YmsLinkDataSiteType.SEARCH, System.currentTimeMillis(),
// linkData.getUrl(), linkData.getKeyword()).execute();
这是引发错误的测试用例(我知道它目前实际上并没有测试任何东西。一旦成功保存到数据库,我就会这样做):
@Test
public void batchInsert() throws InterruptedException, SQLException
int batchCount = 0;
List<YmsLinkDataRecord> batchRecords = Lists.newArrayList();
for (YmsLinkDataRecord ld : ConfigurationToYmsLinkDataRecord.convert(
config))
batchCount++;
batchRecords.add(ld);
if (batchCount == 100)
ldService.create(batchRecords);
batchRecords.clear();
batchCount = 0;
ldService.create(batchRecords);
任何帮助将不胜感激!
【问题讨论】:
请添加插入记录的代码。您是否也考虑过使用 Spring Batch 处理这种情况? 不,实际上我没有。我刚刚接受了 jOOQ 的说法,即 BoneCP 是目前最快的连接池,但我发现缺乏可用信息非常令人沮丧。 我改用了 Apache Commons DBCP,它运行良好。非常感谢您的帮助,我真的很感激。 顺便说一句:“我刚刚接受了 jOOQ 的说法,即 BoneCP 是目前最快的连接池” 我不认为我们这么说......我们实际上只是接受了any 用于我们的 jOOQ/Spring 教程的连接池 :-)(当然,这对您的问题没有帮助...)鉴于您在 BoneCP 与 DBCP 方面的经验,我认为我们可能会切换为好吧。感谢您的耐心等待,对于给您带来的不便,我们深表歉意。 @TheShahin:不客气。感谢像您这样的***用户,jOOQ 社区正在不断发展壮大。 [Petri Kainulainen](www.petrikainulainen.net/tag/jooq/) 目前正在编写一套优秀的教程(我们的 Spring 教程基于此)。如果您发布某些内容,我们肯定会在我们的博客/Twitter/Community 网站上为您推荐。我们还为杰出贡献发送stickers :-) 【参考方案1】:BoneCP 有一个相当有趣的“功能”:如果查询失败并出现“致命”错误代码池将关闭所有连接并变得不可用。据我记得,当 MySQL 由于缺少列之类的原因而出现“HY00”错误时,我遇到了类似的问题
相关代码:https://github.com/wwadge/bonecp/blob/master/bonecp/src/main/java/com/jolbox/bonecp/ConnectionHandle.java#L182
好像“HY00”在最新版本中不再被视为faatl
【讨论】:
这为我节省了很多时间来思考为什么 Ingres 连接在第一次获取时都“关闭”。请注意,BoneCP 也不适合 Ingres。以上是关于BoneCP 抛出“SQLException:连接已关闭!”批量插入 MySQL 时的主要内容,如果未能解决你的问题,请参考以下文章
java.sql.SQLException:连接已关闭 [POOL-HikariCP]