点燃扫描查询问题

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了点燃扫描查询问题相关的知识,希望对你有一定的参考价值。

我使用Ignite将K-V类型为String-String的缓存值存储在名为“ DOCIDS”的缓存中。

我正在使用HashMap的二进制转换对象作为另一个名为“ URLS”的K-V缓存中的值。密钥将仅是字符串,但值将为“ com.xyz.scrapy.common.ignite.IgniteObject”。

现在,当我查询扫描操作时,我正在这样做

//This scan query should only happen on cache which has K-V types as String-String.
        public Object scan(Object... allObjects) throws Exception 
            Set<Object> matchedKeys = new HashSet<>();
            try (QueryCursor<Cache.Entry<String, String>> cursor =
                         cacheHolder.getSecondaryCache().query(new ScanQuery<>(
                                 (k, v) -> k.contains(allObjects[0].toString())
                         ))) 
                for (Cache.Entry<String, String> entry : cursor) 
                    matchedKeys.add(entry.getKey());
                
            
            return matchedKeys;
        

在allObjects中,该值每次都是某种String模式。那是肯定的。但是,当Ignite筛选这些值时,某个地方的值就是IgniteObject类型,而我并不是首先将其插入“ DOCIDS”缓存中。异常附在下面:

"com.xyz.scrapy.common.ignite.ScrapyIgniteException: IGNITE EXCEPTION :: Exception while scanning for keys with given pattern in Ignite db - DOCIDS. Exception - IDSCaused by: javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to execute query on node [query=GridCacheQueryBean [qry=GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, filter=com.xyz.scrapy.common.ignite.IgniteQuery$QueryObject$$Lambda$243/2029270129@58665d95, transform=null, part=null, incMeta=false, pageSize=1024, timeout=0, incBackups=false, forceLocal=false, dedup=false, prj=org.apache.ignite.internal.cluster.ClusterGroupAdapter@1dfc48e7, keepBinary=false, subjId=0ef0942f-d2d1-4ac0-aa5b-783b3e1a0960, taskHash=0, mvccSnapshot=null, dataPageScanEnabled=null], rdc=null, trans=null], nodeId=f01b04de-66c0-4edc-8efe-4977ecf656b1]
    at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272)
    at org.apache.ignite.internal.processors.cache.query.GridCacheQueryFutureAdapter.next(GridCacheQueryFutureAdapter.java:167)
    at org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$5.onHasNext(GridCacheDistributedQueryManager.java:645)
    at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:52)
    at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:44)
    at org.apache.ignite.internal.processors.cache.AutoClosableCursorIterator.hasNext(AutoClosableCursorIterator.java:47)
    at com.xyz.scrapy.common.ignite.IgniteQuery$QueryObject.scan(IgniteQuery.java:103)
    at com.xyz.scrapy.common.ignite.IgniteQuery.executeFieldsQuery(IgniteQuery.java:47)
    at com.xyz.scrapy.common.ignite.IgniteConnectionHandler$CacheHolder.scanKeys(IgniteConnectionHandler.java:302)
Caused by: class org.apache.ignite.binary.BinaryInvalidTypeException: com.xyz.scrapy.common.ignite.IgniteObject
    at org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:643)
    at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1755)
    at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
    at org.apache.ignite.internal.processors.cache.CacheObjectUtils.unwrapBinary(CacheObjectUtils.java:177)
    at org.apache.ignite.internal.processors.cache.CacheObjectUtils.unwrapBinaryIfNeeded(CacheObjectUtils.java:40)
    at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3050)
    at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2952)
    at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:52)
    at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:44)
    at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1141)
    at org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:232)
    at org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:108)
    at org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:106)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1635)
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1255)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4300(GridIoManager.java:144)
    at org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1144)
 Caused by: java.lang.ClassNotFoundException: com.xyz.scrapy.common.ignite.IgniteObject
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

有人可以解释为什么Ignite从缓存DOCIDS获取IgniteObject吗?因为我只将IgniteObject存储在“ URLS”缓存中,而不存储在“ DOCIDS”中。实际的例外是它没有加载IgniteObject类。我什至不知道为什么它不能加载,因为我可以插入那些对象而没有任何问题。

IgniteObject类如下

import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.binary.BinaryObjectBuilder;
import java.io.Serializable;

//If value is not String, every value of K-V Store should wrap its contents with this Object
public class IgniteObject implements Serializable 

    //Value set via Reflection
    private Object object;

    public static BinaryObject convertToBinaryObject(Object object) throws Exception
        //IgniteConnectionHandler.getConnection() will get Ignite Object named ignite. getObjectBuilder will do ignite.binary().builder(stringValue)
        BinaryObjectBuilder builder = IgniteConnectionHandler.getConnection().getObjectBuilder(IgniteObject.class.getName());
        builder.setField("object", object);//No I18N
        return builder.build();
    

    public Object getActualObject()
        return object;
    

注意:cacheHolder.getSecondaryCache()将提供对象-IgniteCache。

答案

即使您仅使用密钥,也会在Cache.Entry中同时获得密钥和值,这很容易发生。

您可以尝试

cacheHolder.getSecondaryCache().withKeepBinary().query(...)

为了避免需要IgniteObject类。

以上是关于点燃扫描查询问题的主要内容,如果未能解决你的问题,请参考以下文章

DynamoDb:扫描查询不会返回所有数据

优化 SQL 查询以避免全表扫描

Redshift Spectrum 为查询扫描的数据大小

SQL Server 表扫描时间是不是取决于查询?

优化 MySQL 查询以避免扫描大量行

SQL中的表扫描和索引扫描