Java操作Elasticsearch6实现count统计distinct去重

Posted 2022-11-27 好好生活_

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Java操作Elasticsearch6实现count统计distinct去重相关的知识，希望对你有一定的参考价值。

引言

很久没有更新Elasticsearch系列文章，一方面是之前学会的条件查询足以满足项目需求，另一方面是前段时间一直很忙，几个项目的需求交叉进行，没什么时间学习新的东西。

本篇博客将更新count、distinct、count(distinct)这几个新学到的Elasticsearch关于查询的方法。

1. Count计算

在项目中，count也算是一个比较常用的方法。之前项目中有需要用到，都是基于查询所有的方法，直接取total的值返回。最近发现，其实Elasticsearch单独封装了一个CountRequest类来获取数量。代码如下：

/**
 * 查询指定索引文档总数（可增加查询条件，如果为空，则查询所有）
 */
@Test
public void testCount() 
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    CountRequest countRequest = new CountRequest();
    //构造查询条件
//        searchSourceBuilder.query(QueryBuilders.termQuery("fieldName", 1));
    countRequest.indices("indexName").source(searchSourceBuilder);
    CountResponse countResponse = null;
    try 
        countResponse = restHighLevelClient.count(countRequest, RequestOptions.DEFAULT);
        return countResponse.getCount();
     catch (IOException e) 
        log.error("[EsClientConfig.countDocumentSize][error][fail to count document size,param is ]", countRequest);
    
    return 0;
    log.info("[document size is , indexName is ]", size, indexName);

2. distinct查询

distinct同样属于项目中使用较广泛的查询方法，对应mysql中的sql语句如下：

select distinct(field_name) from table_name;

下面就来看下Elasticsearch6中，我们该如何实现，代码如下：

/**
 * 查询指定索引下，按某个字段distinct的结果
 * 需要注意的：
 * 1. 如果字段是keyword类型，需要在字段后加上keyword，形如：subject.keyword
 * 2. 不指定size的情况下，默认只返回10条
 */
@Test
public void testDistinct() 
    SearchRequest searchRequest = new SearchRequest();
    searchRequest.indices("indexName").types("indexType");
    BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
    //构造条件
//        boolQueryBuilder
//                .filter(QueryBuilders.termQuery("fieldName", "fieldValue"));
	//指定distinct字段
    CollapseBuilder collapseBuilder = new CollapseBuilder("fieldName.keyword");
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(boolQueryBuilder).collapse(collapseBuilder);
    //执行查询
    searchRequest.source(searchSourceBuilder);
    String key = "fieldName.keyword";
    List<Object> resultList = new ArrayList<>();
    SearchResponse response;
    try 
        response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits hits = response.getHits();
        if (hits.getTotalHits() > 0) 
            SearchHit[] searchHitsArr = hits.getHits();
            for (int i = 0; i < searchHitsArr.length; i++) 
            //在Fields属性中可以获取对应字段的值,这里我增加了一个参数key对应distinct的字段名
            	resultList.add(searchHitsArr[i].getFields().get(key).getValue());
            
        
     catch (IOException e) 
        log.error("[EsClientConfig.searchDocumentDistinct] [error] [fail to query, param is ,key is ]", JSON.toJSON(searchRequest), key, e);
        return resultList;
    
    return resultList;

3. Count(distinct)查询

先拿项目中的场景举个例子，学生可以报名参加多门学科的多场公开课，现在想要知道某个学生参加了几门学科的公开课，对应mysql中的sql语句如下：

select count(distinct(field_name)) from table_name where student_id = '**';

对应Elasticsearch中的查询，代码如下：


/**
 * 计算指定索引，按某个字段去重后的count结果
 */
@Test
public void testCountDistinct() 
    SearchRequest searchRequest = new SearchRequest();
    searchRequest.indices("indexName").types("indexType");
    String key = "subjectId";
    BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
    //查询条件，查询指定学员
    boolQueryBuilder
            .filter(QueryBuilders.termQuery("uuid", "**"));
    //指定count(distinct)字段名,cardinality为指定字段的别名,field为指定字段
    CardinalityAggregationBuilder aggregationBuilder = AggregationBuilders.cardinality(key).field("subjectId.keyword");
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(boolQueryBuilder).aggregation(aggregationBuilder);
    //执行查询
    searchRequest.source(searchSourceBuilder);
    int count = 0;
    SearchResponse response;
    try 
        response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
        ParsedCardinality result = response.getAggregations().get(key);
        log.info("result is ", result.getValue());
     catch (IOException e) 
        log.error("[EsClientConfig.countDocumentDistinct] [error] [fail to query, param is ]", JSON.toJSON(searchRequest), e);
        return count;
    
    return count;