在进行大量写入时提高 neo4j 性能

Posted 2023-04-12

【中文标题】在进行大量写入时提高 neo4j 性能

【英文标题】：Increasing neo4j performance when doing large number of writes

【发布时间】：2018-03-29 23:07:26

【问题描述】：

我正在尝试通过我的应用程序将数据加载到 neo4j。这是使用螺栓驱动器。我的应用程序最初是写密集型的。我无法减少加载时间，因为 neo4j 使用螺栓驱动程序写入似乎很慢。我看到所有线程都在执行以下操作：

at sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
  at sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
  at sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher)
  at sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher)
  at sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer)
  at org.neo4j.driver.internal.security.TLSSocketChannel.channelRead(java.nio.ByteBuffer) (line: 159)
  at org.neo4j.driver.internal.security.TLSSocketChannel.unwrap(java.nio.ByteBuffer) (line: 229)
  at org.neo4j.driver.internal.security.TLSSocketChannel.read(java.nio.ByteBuffer) (line: 419)
  at org.neo4j.driver.internal.net.BufferingChunkedInput.readNextPacket(java.nio.channels.ReadableByteChannel, java.nio.ByteBuffer) (line: 409)
  at org.neo4j.driver.internal.net.BufferingChunkedInput.readChunkSize() (line: 345)
  at org.neo4j.driver.internal.net.BufferingChunkedInput.read(java.nio.ByteBuffer) (line: 247)
  at org.neo4j.driver.internal.net.BufferingChunkedInput.fillScratchBuffer(int) (line: 216)
  at org.neo4j.driver.internal.net.BufferingChunkedInput.readByte() (line: 110)
  at org.neo4j.driver.internal.packstream.PackStream$Unpacker.unpackStructHeader() (line: 430)
  at org.neo4j.driver.internal.messaging.PackStreamMessageFormatV1$Reader.read(org.neo4j.driver.internal.messaging.MessageHandler) (line: 398)
  at org.neo4j.driver.internal.net.SocketClient.receiveOne(org.neo4j.driver.internal.net.SocketResponseHandler) (line: 176)
  at org.neo4j.driver.internal.net.SocketConnection.receiveOne() (line: 212)
  at org.neo4j.driver.internal.net.ConcurrencyGuardingConnection.receiveOne() (line: 165)
  at org.neo4j.driver.internal.net.pooling.PooledSocketConnection.receiveOne() (line: 183)
  at org.neo4j.driver.internal.InternalStatementResult.receiveOne() (line: 335)
  at org.neo4j.driver.internal.InternalStatementResult.tryFetchNext() (line: 325)
  at org.neo4j.driver.internal.InternalStatementResult.hasNext() (line: 193)

堆内存最小值和最大值 =4Gb 页面缓存大小 = 2 GB 数据库总大小约为 2 GB（预计增长到 10GB）

有什么方法可以优化上述操作吗？我可以将堆增加到 8 GB，因为其他应用程序也在这台机器上运行。

【问题讨论】：

您需要出示您的代码。

【参考方案1】：

如果您的应用程序是写密集型的，您将受益于使用批处理事务执行写入（从您的应用程序执行此操作 -> 聚合节点 \ 边缘直到您到达这一边，打开一个事务，并在何时提交达到这个尺寸）。您可以使用批量大小（1000、10000），看看什么最适合您。