将 Big Derby 嵌入数据库迁移到 HSQLDB 抛出 java.lang.OutOfMemoryError: Java heap space

Posted 2023-03-14

【中文标题】将 Big Derby 嵌入数据库迁移到 HSQLDB 抛出 java.lang.OutOfMemoryError: Java heap space

【英文标题】：Migrating a Big Derby embebed database to HSQLDB throw java.lang.OutOfMemoryError: Java heap space

【发布时间】：2019-04-11 12:38:20

【问题描述】：

我正在尝试在 Spring Boot 服务中将大数据库从 derby 迁移到 HSQLDB，例如 10 列的几个表中的 1.5M regs。我正在检查 VisualVM； byte 和 char 消耗大量内存。但时间上最大的差异是在德比比赛中。

有时会在此处抛出错误，但有时会在其他控制器中抛出错误。我不想触摸所有文件来添加我的 catchOutofMemory 以重新启动。

以下是我的代码的一个版本，块注释显示了进程的简历：

run()//thread inside static function.
    while(keepMigrating)
        keepMigrating=Migrate();
    

private static boolean Migrate(JdbcTemplate derby,JdbcTemplate hsql)
    int regs = 100000;
    PreparedStatement statement = null;
    ResultSet rs = null;
    PreparedStatement statementHSQL = null;
    try 
        for (String table : tables) //tables contains all tables to migrate
        //check how many registers left asd asign to cant, if cant is 0 the empty is true.
        PreparedStatement statementUpd[];
            while (!empty) 
                if (trys <= 0) throw new Exception("redo");
                //check how many registers left asd asign to cant, if cant is 0 the empty is true and out of bucle and ready to check next table
                /*
                *Next process resume as:
                *fetch data from derby that hasnt been migrated limited by cant
                *create a batch to insert in hsql
                *create a update for derby
                *create a delete in case someting goes wrong
                *excecute insert and update, if someting in batch fail delete the entry in migrate table
                *reduce regs to get out of migrate method at some ponint.
                */
                statement = derby.getDataSource().getConnection().prepareStatement(
                MessageFormat.format(select_all_migrate_false_and_fetch_cant,table));
                statementUpd = new PreparedStatement[cant];
                ArrayList<String> deleteIds = new ArrayList<>();
                StringBuilder columnNames = new StringBuilder();
                StringBuilder updateSQL = new StringBuilder();
                StringBuilder bindVariables = new StringBuilder();
                try 
                    ResultSetMetaData meta = rs.getMetaData();
                    for (int i = 1; i <= meta.getColumnCount(); i++) 
                        if (!meta.getColumnName(i).equals("MIGRATED")) 
                            if (i > 1) 
                                columnNames.append(", ");
                                bindVariables.append(", ");
                            
                            columnNames.append(meta.getColumnName(i));
                            bindVariables.append('?');
                        
                    
                    String sql = "INSERT INTO " + table.substring(4) + " ("
                            + columnNames
                            + ") VALUES ("
                            + bindVariables
                            + ")";
                    statementHSQL = hsql.getDataSource().getConnection().prepareStatement(sql);
                    HashMap<String, Object> data = new HashMap<>();
                    int row = 0;
                    int lastId = 0;
                    String columnName;
                    while (rs.next()) 
                        for (int i = 1; i <= meta.getColumnCount(); i++) 
                            columnName = meta.getColumnName(i);
                            Object o = rs.getObject(i);
                            statementHSQL.setObject(i, o);
                            if (columnName.equals(mainColumn))
                                deleteIds.add(String.valueOf(o));
                            if (!(meta.getColumnType(i) == 2004)) data.put(columnName, o);
                            if (columnName.equals(mainColumn)) id = rs.getObject(i);
                        
                        int c = 1;
                        String update = MessageFormat.format("INSERT INTO 0M (1M, MIGRATED) VALUES(?, TRUE)",
                                table.substring(4), mainColumn).replace("\"M", "M\"");//migrated state is saved in other table
                        lastId = Integer.valueOf(String.valueOf(id));
                        statementUpd[row] = derby.getDataSource().getConnection().prepareStatement(update);
                        statementUpd[row].setObject(1, rs.getObject(mainColumn));
                        updateSQL = new StringBuilder();
                        statementHSQL.addBatch();
                        row += 1;
                    
                    /*
                     * Build delete query in case of inserted values in HSQLDB but not updated in DERBY
                     */
                    StringBuilder builder = new StringBuilder();
                    builder.append("(");
                    int count = 1;
                    for (String s : deleteIds) 
                        if (count > 1) builder.append(", ");
                        builder.append("?");
                        count++;
                    
                    builder.append(")");
                    String str = builder.toString();
                    String queryDelete = "DELETE FROM " + table.substring(4) + " WHERE " + mainColumn + " IN " + str;
                    PreparedStatement statementHSQLDel = hsql.getDataSource().getConnection().prepareStatement
                            (queryDelete);
                    int c = 1;
                    for (String s : deleteIds) 
                        statementHSQLDel.setObject(c, s);
                        c++;
                    
                    boolean deletes = statementHSQLDel.execute();
                    statementHSQLDel.close();
                    try 
                        DatabaseUtils.close(statementHSQLDel);
                     catch (Exception e) 
                        catchOutOfMemory(e);
                    
                    int[] result = statementHSQL.executeBatch();
                    StringBuilder resultS = new StringBuilder();
                    int stCounter = 0;
                    int stCounterInsert = 0;
                    int stCounterUpdate = 0;
                    String notarydebug;
                    for (int i : result) 
                        int upd = 0;
                        try 
                            if (i == 1) upd = statementUpd[stCounter].executeUpdate();
                         catch (Exception e) 
                            catchOutOfMemory(e);
                        
                        stCounterInsert += i;
                        stCounterUpdate += upd;
                        resultS.append(",").append(String.valueOf(i)).append("-").append(String.valueOf(upd));
                        stCounter += 1;
                    
                    statementHSQL.clearBatch();
                    try 
                        DatabaseUtils.close(statementHSQL);
                     catch (Exception e) 
                        catchOutOfMemory(e);
                    
                 catch (SQLException se) 
                    catchOutOfMemory(se);//otherstuff
                 catch (Exception e) 
                    catchOutOfMemory(e);
                
                try 
                    DatabaseUtils.close(rs);
                    DatabaseUtils.close(statement);
                 catch (Exception e) 
                    catchOutOfMemory(e);
                
                regs=regs-cant;
            
        
    catch (Exception e) 
        if (e.getMessage().equals("redo")) return true;//end the loop of regs maximun and get out of method.
    
return false;//end migration succesfully

private static int catchOutOfMemory(Throwable e) 
    if (e == null) return 0;
    if (e instanceof OutOfMemoryError) 
        Application.restartBat();
        return 1;
     else 
        return catchOutOfMemory(e.getCause());
    

编辑： 所以我按照评论中的建议进行更改以接受提交，如下所示：

Connection hsqlCon;
PrepareStatement hsqlStm;
hsqlCon = JdbcHSQLDB.getDataSource().getConnection();
hsqlStm = hsqlCon.prepareStatement(sql);
hsqlStm.addBatch();
hsqlStm.execute();
hsqlStm.close();
hsqlCon.close();

但我得到了相同的堆内存消耗：

【问题讨论】：

HSQLDB 使用的 URL 是什么？您是否使用内存中的 URL 而不是基于文件的 URL？

bot DERBY 和 HSQLDB 是文件

我看到的唯一另一个问题是，当你使用它们时，你没有正确关闭连接和语句（尽管你的代码很难遵循它的结构），这会导致内存中的更多对象不必要的。

谢谢@Mark，如果语句保持打开状态，我会一一再次检查，我在某些地方使用了 null 并找到了一个。

你需要在多个事务中进行这种传输并提交每个事务。 HSQLDB 将未提交的行保存在内存中，如果数据库很大，可能会耗尽。

【参考方案1】：

HSQLDB 中的表类型在提供的代码中并不清楚。您必须对每个表使用此语句一次，以确保表数据存储在finename.data 文件中：

SET TABLE tableName TYPE CACHED

批量 INSERT 报告的顺序不正确。使用这个序列：

Connection hsqlCon;
PrepareStatement hsqlStm;
hsqlCon = JdbcHSQLDB.getDataSource().getConnection();
hsqlStm = hsqlCon.prepareStatement(sql);

 // repeat this block until all is finished
     // repeat for 1000 rows
        hsqlStm.addBatch();
     
    hsqlStm.executeBatch(); // after every 1000 rows
 

hsqlStm.close();
hsqlCon.close();