Apache Ignite 频繁缓存关闭异常
Posted
技术标签:
【中文标题】Apache Ignite 频繁缓存关闭异常【英文标题】：Apache Ignite Frequent Cache Close Exception 【发布时间】：2020-02-20 12:05:23 【问题描述】：我们使用 Apache Ignite 作为缓存来加速我们的授权和权限调用。在客户端上的应用程序加载期间，应用程序通过 get 调用访问 Ignite 缓存大约 18 次，在此期间，我们经常收到来自 Ignite 的缓存关闭异常。我们一直在尝试通过大量调用来复制相同的内容，但是在本地运行时，每 18 个调用中的每 3/4 个调用似乎都会发生错误
我们已将所有推荐的配置应用到集群
<!-- Alter configuration below as needed. -->
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="true" />
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default" />
<property name="atomicityMode" value="ATOMIC" />
<property name="backups" value="1" />
</bean>
</list>
</property>
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED" />
</list>
</property>
<property name="binaryConfiguration">
<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="compactFooter" value="false" />
</bean>
</property>
<!-- Configure internal thread pool. -->
<property name="publicThreadPoolSize" value="64" />
<!-- Configure system thread pool. -->
<property name="systemThreadPoolSize" value="32" />
<!--<property name="clientMode" value="false" />-->
<property name="sqlSchemas">
<list>
<value>BA_EV</value>
<value>BA_DEMO</value>
<value>TEST_1</value>
<value>TEST_2</value>
<value>TEST_3</value>
</list>
</property>
<!-- Enabling Apache Ignite Persistent Store. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true" />
</bean>
</property>
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="slowClientQueueLimit" value="1000" />
<property name="messageQueueLimit" value="1024" />
</bean>
</property>
<!-- Enabling authentication. -->
<property name="authenticationEnabled" value="true" />
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>localhost:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
我们使用的是简单的 cache.get() 和 cache.put() 方法，它们是通过注解调用的。
目前我们正在使用
try
cache.put(key, returnType.cast(result));
catch (CacheException | org.hibernate.cache.CacheException e)
if (e.getCause() instanceof IgniteClientDisconnectedException)
IgniteClientDisconnectedException cause =(IgniteClientDisconnectedException)e.getCause();
cause.reconnectFuture().get(); // Wait for reconnection.
cache.put(key, returnType.cast(result));
addDiconnectCount();
LOGGER.error("Diconnection Reason Trace: ", e);
这可以解决错误，但我们无法找出频繁重新连接缓存的根本原因。我们在本地运行 4GB XMX，在 ignite 集群和客户端上大约有 86% 未使用的堆内存和可忽略的节点。
我们试图存储的对象有以下类型：
Map<String, Serializable>
map.put(String, String[]);
List<String>
日志：
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.7.6#20190911-sha1:21f7ca41c4348909e2fd26ccf59b5b2ce1f4474e
>>> +----------------------------------------------------------------------+
>>> OS name: Windows 10 10.0 amd64
>>> CPU(s): 4
>>> Heap: 2.0GB
>>> VM name: 4484@xxxxxxx
>>> Local node [ID=7F470894-A979-443C-BC4F-7BCB047C7550, order=6, clientMode=true]
>>> Local node addresses: [xxxxxxx.xx.xxxxxxx.com/0:0:0:0:0:0:0:1, BLREQX1352123L.xx.xxxxxxx.com/10.73.4.44, 192.168.138.1/127.0.0.1, BLREQX1352123L.xx.xxxxxxx.com/172.2 2.192.1, /192.168.138.1, /192.168.234.1]
>>> Local ports: TCP:10801 TCP:47101
[16:39:14,562][INFO][Thread-10][GridDiscoveryManager] Topology snapshot [ver=6, locNode=7f470894, servers=1, clients=1, state=ACTIVE, CPUs=4, offheap=3.2GB, heap=4.0GB] [16:39:45,598][INFO][main][Http11NioProtocol] Starting ProtocolHandler ["http-nio-5014"]
[16:39:45,598][INFO][main][NioselectorPool] Using a shared selector for servlet write/read
[16:40:14,564][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7f470894, uptime=00:01:00.016]
^-- H/N/C [hosts=1, nodes=2, CPUs=4]
^-- CPU [cur=0.13%, avg=31.59%, GC=0%]
^-- PageMemory [pages=0]
^-- Heap [used=183MB, free=91.06%, comm=1024MB]
^-- Off-heap [used=0MB, free=-1%, comm=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=0, qSize=0]
[16:40:20,864][INFO][exchange-worker-#38][GridCacheProcessor] Started cache [name=userRoleCache, id=-2021367519, memoryPolicyName=null, mode=REPLICATED, atomicity=ATOMIC, ba ckups=2147483647, mvcc=false], encryptionEnabled=false]
[16:40:20,888][INFO][exchange-worker-#38][GridCacheProcessor] Finish proxy initialization, cacheName=userRoleCache, localNodeId=7f470894-a979-443c-bc4f-7bcb047c7550
[16:40:20,997][INFO][exchange-worker-#38][GridCacheProcessor] Stopped cache [cacheName=userRoleCache]
[16:40:21,000][INFO][exchange-worker-#38][GridCacheProcessor] Can not finish proxy initialization because proxy does not exist, cacheName=userRoleCache, localNodeId=7f470894 -a979-443c-bc4f-7bcb047c7550
[16:40:21,001][INFO][exchange-worker-#38][GridCacheProcessor] Can not finish proxy initialization because proxy does not exist, cacheName=userRoleCache, localNodeId=7f470894 -a979-443c-bc4f-7bcb047c7550
[16:40:21,001][INFO][exchange-worker-#38][GridCacheProcessor] Can not finish proxy initialization because proxy does not exist, cacheName=userRoleCache, localNodeId=7f470894 -a979-443c-bc4f-7bcb047c7550
[16:40:21,015][INFO][exchange-worker-#38][GridCacheProcessor] Started cache [name=roleAppPermissions, id=-686786119, memoryPolicyName=null, mode=REPLICATED, atomicity=ATOMIC , backups=2147483647, mvcc=false], encryptionEnabled=false]
[16:40:21,039][INFO][exchange-worker-#38][GridCacheProcessor] Finish proxy initialization, cacheName=roleAppPermissions, localNodeId=7f470894-a979-443c-bc4f-7bcb047c7550 [16:40:21,063][INFO][exchange-worker-#38][GridCacheProcessor] Stopped cache [cacheName=roleAppPermissions]
[16:40:21,131][SEVERE][http-nio-5014-exec-4][JerseyConfig]] Servlet.service() for servlet [com.xxxx.xxxx.base.config.JerseyConfig] in context with path [/ba-cct-api] threw e xception
【问题讨论】：
不看日志很难说这里发生了什么。您的日志中是否有 JVM 暂停消息？可以分享一下吗？ 已添加日志，请检查 日志看起来不错，你能显示你正在捕获的异常的详细信息吗？ 剩下的就是一个简单的异常堆栈传播。除了消息缓存已关闭外，没有什么其他内容。我添加了异常检查以在重新连接后执行操作。如有必要，我可以添加这些日志，但它们无话可说。 您能否提供完整的日志并将 IGNITE_QUIET 设置为 false？也许有一些关于重新连接的原因。 【参考方案1】：我终于能够弄清楚这一点，并追溯到我们的自定义包装器，其中一个地方有一个 try-with-resources 逻辑。由于 Apache Ignite 每次都重新打开缓存连接，因此只有在并发命中时我们才能看到此错误。
谢谢大家的回答。
【讨论】：
以上是关于Apache Ignite 频繁缓存关闭异常的主要内容，如果未能解决你的问题，请参考以下文章