KETTLE_内存溢出错误

Posted 2022-12-16 黄炎龙

原创作品，出自 “深蓝的blog” 博客，欢迎转载，转载时请务必注明以下出处，否则追究版权法律责任。

深蓝的blog：http://blog.csdn.net/huangyanlong/article/details/42453831

 

kettle内存溢出错误解决

环境：
源端数据库：oracle 10G R2
目标端数据库：oracle 11G R2
kettle版本：5.0.1-stable

报错：
抽取大数据量表时，报错，日志信息如下：

2015/01/05 11:27:42 - Spoon - 转换已经打开.
2015/01/05 11:27:42 - Spoon - 正在打开转换 [TABLE_BWWN]...
2015/01/05 11:27:42 - Spoon - 开始执行转换.
2015/01/05 11:27:42 - TABLE_BWWN - 为了转换解除补丁开始  [TABLE_BWWN]
2015/01/05 11:27:44 - max.0 - Finished reading query, closing connection.
2015/01/05 11:27:44 - max.0 - 完成处理 (I=1, O=0, R=0, W=1, U=0, E=0
2015/01/05 12:25:08 - 插入 / 更新.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : UnexpectedError:
2015/01/05 12:25:08 - Spoon - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 一个未预期的错误发生在Spoon: probable cause:在停止Spoon前，请先关闭其它spoon窗口! 
2015/01/05 12:25:08 - Spoon - Java heap space
2015/01/05 12:25:08 - 插入 / 更新.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: Java heap space
2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatement.bindBytesAsStream(OraclePreparedStatement.java:6713)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatement.setBytesInternal(OraclePreparedStatement.java:6700)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatement.setBytes(OraclePreparedStatement.java:6660)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatementWrapper.setBytes(OraclePreparedStatementWrapper.java:150)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.row.value.ValueMetaBase.setPreparedStatementValue(ValueMetaBase.java:4704)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.database.Database.setValue(Database.java:928)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.database.Database.setValues(Database.java:943)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.database.Database.setValuesInsert(Database.java:888)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues(InsertUpdate.java:113)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:301)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60)
2015/01/05 12:25:08 - 插入 / 更新.0 -  at java.lang.Thread.run(Unknown Source)
2015/01/05 12:25:08 - Spoon - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: Java heap space
2015/01/05 12:25:08 - Spoon -  at java.util.Arrays.copyOf(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.safeTrim(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.access$100(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.decode(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.decode(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.String.<init>(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at java.lang.String.<init>(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.core.row.value.ValueMetaBase.convertBinaryStringToString(ValueMetaBase.java:1105)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.core.row.value.ValueMetaBase.getString(ValueMetaBase.java:1452)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.core.row.RowMeta.getString(RowMeta.java:230)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.showPreviewGrid(TransPreviewDelegate.java:302)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.refreshView(TransPreviewDelegate.java:268)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate$1.widgetSelected(TransPreviewDelegate.java:188)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.notifyListeners(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.custom.CTabFolder.setSelection(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.custom.CTabFolder.onMouse(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.custom.CTabFolder$1.handleEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1227)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7368)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:8673)
2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:625)
2015/01/05 12:25:08 - Spoon -  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015/01/05 12:25:09 - 插入 / 更新.0 - 完成处理 (I=20575, O=20574, R=20575, W=20574, U=0, E=1
2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN
2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN
2015/01/05 12:25:09 - 表输入.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Unexpected error
2015/01/05 12:25:09 - 表输入.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException: 
2015/01/05 12:25:09 - 表输入.0 - Couldn't get row from result set
2015/01/05 12:25:09 - 表输入.0 - ORA-03111: 通信通道收到中断

2015/01/05 12:25:09 - 表输入.0 - 
2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.core.database.Database.getRow(Database.java:2302)
2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.core.database.Database.getRow(Database.java:2270)
2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:153)
2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60)
2015/01/05 12:25:09 - 表输入.0 -  at java.lang.Thread.run(Unknown Source)
2015/01/05 12:25:09 - 表输入.0 - Caused by: java.sql.SQLException: ORA-03111: 通信通道收到中断

2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1082)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:369)
2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:273)
2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.core.database.Database.getRow(Database.java:2290)
2015/01/05 12:25:09 - 表输入.0 -  ... 4 more
2015/01/05 12:25:09 - 表输入.0 - Finished reading query, closing connection.
2015/01/05 12:25:09 - 表输入.0 - 完成处理 (I=20580, O=0, R=1, W=20579, U=0, E=1
2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到!
2015/01/05 12:25:09 - Spoon - 转换完成!!
2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到!
2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到!
2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN
2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN

解决方法：
找到kettle目录下的Spoon.bat文件，找到如下部分：

REM ******************************************************************
REM ** Set java runtime options                                     **
REM ** Change 512m to higher values in case you run out of memory   **
REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable      **
REM ******************************************************************

if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx512m" "-XX:MaxPermSize=256m"

set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"

修改参数，将内存值调大（256的倍数），调整后如下所示：

REM ******************************************************************
REM ** Set java runtime options                                     **
REM ** Change 512m to higher values in case you run out of memory   **
REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable      **
REM ******************************************************************

if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx1024m" "-XX:MaxPermSize=2048m"

set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"

重新执行kettle抽取数据，不再报内存溢出错误。

小结：
        使用kettle时，如果遇到数据量大、大对象时很容易出现内存溢出现象，此时需要根据实际的硬件配置对预设的java堆栈大小进行调整。

 

原创作品，出自 “深蓝的blog” 博客，欢迎转载，转载时请务必注明以下出处，否则追究版权法律责任。

深蓝的blog：http://blog.csdn.net/huangyanlong/article/details/42453831