Apache Ignite 2.x - 我可以利用堆外存储并且仍然有基于时间的驱逐(从堆上和堆外存储)吗?
Posted
技术标签:
【中文标题】Apache Ignite 2.x - 我可以利用堆外存储并且仍然有基于时间的驱逐(从堆上和堆外存储)吗?【英文标题】:Apache Ignite 2.x - can I take advantage of offheap storage and also still have time based eviction (from the on & off heap storage)? 【发布时间】:2021-02-03 22:55:56 【问题描述】:我正在使用 Apache Ignite 2.8.1
我有一个特定的缓存需要在堆外存储(因为否则我会受到内存不足的困扰)。而且我还需要分发缓存。
但我还想要一个基于时间的驱逐策略来从开/关内存缓存中驱逐条目。
Apache Ignite 2.x 甚至可以做到这一点吗?
在此处查看 sn-p。我尝试过以各种方式进行配置,但除了手动操作之外,没有任何方法可以从缓存中清除这些条目。
当我运行下面的test
方法时,所有条目都保留在缓存中。
import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.DataRegionConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.configuration.NearCacheConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
public class IgniteCache
private static final String ORG_CACHE = IgniteCache.class.getSimpleName() + "Organizations";
private static Ignite ignite;
private static org.apache.ignite.IgniteCache cache;
public void start()
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
DataStorageConfiguration dsCfg = new DataStorageConfiguration();
DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
dfltDataRegConf.setPersistenceEnabled(true);
dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
dsCfg.setStoragePath("E:\\igniteStorage");
igniteConfiguration.setDataStorageConfiguration(dsCfg);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
ignite = Ignition.start(igniteConfiguration);
ignite.active(true);
CacheConfiguration<Long, X12File> cacheCfg = new CacheConfiguration<>(ORG_CACHE);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setBackups(1);
cacheCfg.setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(50000, 40, 5050)); // this doesn't do anything.
cacheCfg.setNearConfiguration(
new NearCacheConfiguration<Long, X12File>()
.setNearEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1_000_000))); // this doesn't do anything.
cache = ignite.getOrCreateCache(cacheCfg);
for (long i = 0; i < 4_000_000; i++)
if (i > 0 && i % 10_000 == 0)
System.out.println("Done: " + i);
cache.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1))) // this expiry policy doesn't do anything
.put(i, new X12File("x12file" + i, LocalDateTime.now().toString()));
public void test()
System.out.println("Checking if cache entries are being properly evicted ...");
int matches = 0;
for (long i = 0; i < 4_000_000; i++)
if (cache.get(i) != null)
++matches;
System.out.println("Matches: " + matches);
这只是 Apache Ignite 2.x 在我的特定用例中的一个缺点吗?
【问题讨论】:
【参考方案1】:我从您的代码和您的解释中了解到,您似乎对驱逐政策和到期政策有些困惑。
-
逐出策略确定在 RAM 用完时应从 RAM 中删除哪些数据。
用于确定缓存条目 TTL 的过期策略。
在您的情况下,您只需要到期政策。看我的例子:
private static final String ORG_CACHE = IgniteCache.class.getSimpleName() + "Organizations";
public static void main(String[] args) throws Exception
DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
dfltDataRegConf.setPersistenceEnabled(true);
DataStorageConfiguration dsCfg = new DataStorageConfiguration();
dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
dsCfg.setStoragePath("/home/kazakov/tmp");
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsCfg);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
try(Ignite ignite = Ignition.start(igniteConfiguration))
ignite.active(true);
CacheConfiguration<Long, X12File> cacheCfg = new CacheConfiguration<>(ORG_CACHE);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setBackups(1);
cacheCfg.setOnheapCacheEnabled(true);
IgniteCache<Long, X12File> cache = ignite.getOrCreateCache(cacheCfg).withExpiryPolicy(new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1)));
for (long i = 0; i < 4_000_000; i++)
if (i > 0 && i % 10_000 == 0)
System.out.println("Done: " + i);
cache.put(i, new X12File("x12file" + i, LocalDateTime.now().toString()));
Thread.sleep(5000);
int matches = 0;
for (long i = 0; i < 4_000_000; i++)
if (cache.get(i) != null)
++matches;
System.out.println("Matches: " + matches);
【讨论】:
我感谢代码 sn-p 帮助我了解要更改的内容,以及您如何指出我真正想要到期的方式。实际上,我们今天早些时候也得出了同样的结论。【参考方案2】:简而言之:
逐出是基于内存的。删除一些记录,因为我的行太多或占用了太多空间 到期是基于时间的。这条记录在指定的时间段内没有被触及,所以应该删除它所有缓存都是堆外的。您可以为缓存配置the expiry policy,如下所示:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="myCache"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="MINUTES"/>
<constructor-arg value="5"/>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
(near/on-heap 缓存确实做了一些事情。只是不是你所期望的!它们都是 on-heap 和工作 到堆外缓存。)
【讨论】:
是的,我混淆了到期和驱逐之间的区别。感谢您帮助我理解。以上是关于Apache Ignite 2.x - 我可以利用堆外存储并且仍然有基于时间的驱逐(从堆上和堆外存储)吗?的主要内容,如果未能解决你的问题,请参考以下文章