Elasticsearchjava 客户端 获取 termvectors

Posted 2021-05-24 九师兄

1.概述

转载：java 客户端 获取 termvectors

elasticsearch的termvectors包括了term的位置、词频等信息。这些信息用于相应的数据统计或开发其他功能，本文介绍termvecters如何使用，如何通过java客户端获取termvectors相关信息。

要使用termvctor首先要配置mapping中field的"term_vector"属性，默认状态es不开启termvector，因为这样会增加索引的体积，毕竟多存了不少元数据。

PUT test
{
  "mappings": {
    "qa_test": {
      "dynamic": "strict",
      "_all": {
        "enabled": false
      },
      "properties": {
        "question": {
          "properties": {
            "cate": {
              "type": "keyword"
            },
            "desc": {
              "type": "text",
              "store": true,
              "term_vector": "with_positions_offsets_payloads",
              "analyzer": "ik_smart"
            },
            "time": {
              "type": "date",
              "store": true,
              "format": "strict_date_optional_time||epoch_millis||yyyy-MM-dd HH:mm:ss"
            },
            "title": {
              "type": "text",
              "store": true,
              "term_vector": "with_positions_offsets_payloads",
              "analyzer": "ik_smart"
            }
          }
        },
        "updatetime": {
          "type": "date",
          "store": true,
          "format": "strict_date_optional_time||epoch_millis||yyyy-MM-dd HH:mm:ss"
        }
      }
    }
  },
  "settings": {
    "index": {
      "number_of_shards": "1",
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "number_of_replicas": "1"
    }
  }
}

注意示例中的"title"的"term_vector"属性。

接下来为索引创建一条数据

PUT qa_test_02/qa_test/1
{
  "question": {
    "cate": [
      "装修流程",
      "其它"
    ],
    "desc": "筒灯，大洋和索正这两个牌子，哪个好？希望内行的朋友告知一下，谢谢！",
    "time": "2016-07-02 19:59:00",
    "title": "筒灯大洋和索正这两个牌子哪个好"
  },
  "updatetime": 1467503940000
}

下面我们看看这条数据上question.title字段的termvector信息

GET qa_test_02/qa_test/1/_termvectors
{
  "fields": [
    "question.title"
  ],
  "offsets": true,
  "payloads": true,
  "positions": true,
  "term_statistics": true,
  "field_statistics": true
}

结果大概这个样子

{
  "_index": "qa_test_02",
  "_type": "qa_test",
  "_id": "1",
  "_version": 1,
  "found": true,
  "took": 0,
  "term_vectors": {
    "question.title": {
      "field_statistics": {
        "sum_doc_freq": 9,
        "doc_count": 1,
        "sum_ttf": 9
      },
      "terms": {
        "和": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 2,
              "start_offset": 4,
              "end_offset": 5
            }
          ]
        },
        "哪个": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 7,
              "start_offset": 12,
              "end_offset": 14
            }
          ]
        },
        "大洋": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 1,
              "start_offset": 2,
              "end_offset": 4
            }
          ]
        },
        "好": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 8,
              "start_offset": 14,
              "end_offset": 15
            }
          ]
        },
        "正": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 4,
              "start_offset": 6,
              "end_offset": 7
            }
          ]
        },
        "牌子": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 6,
              "start_offset": 10,
              "end_offset": 12
            }
          ]
        },
        "筒灯": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 0,
              "start_offset": 0,
              "end_offset": 2
            }
          ]
        },
        "索": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 3,
              "start_offset": 5,
              "end_offset": 6
            }
          ]
        },
        "这两个": {
          "doc_freq": 1,
          "ttf": 1,
          "term_freq": 1,
          "tokens": [
            {
              "position": 5,
              "start_offset": 7,
              "end_offset": 10
            }
          ]
        }
      }
    }
  }
}

下面我们说说如何通过java代码实现termvector的获取，不说废话直接上代码

  TermVectorsResponse     termVectorResponse = client.prepareTermVectors().setIndex(sourceindexname).setType(sourceindextype)
                .setId(id).setSelectedFields(fieldname).setTermStatistics(true).execute()
                .actionGet();
        XContentBuilder builder = XContentFactory.contentBuilder(XContentType.JSON);
        termVectorResponse.toXContent(builder, null);
        System.out.println(builder.string());
        Fields fields = termVectorResponse.getFields();
        Iterator<String> iterator = fields.iterator();
        while (iterator.hasNext()) {
            String field = iterator.next();
            Terms terms = fields.terms(field);
            TermsEnum termsEnum = terms.iterator();
            while (termsEnum.next() != null) {
                BytesRef term = termsEnum.term();
                if (term != null) {
                    System.out.println(term.utf8ToString() + termsEnum.totalTermFreq());
                }
            }
        }

获取TermVectorsResponse的代码很好理解，主要是设置索引名称、索引type、索引id以及需要展示的若干属性。

接下来是如何获取某一term的termvector，有两种方案第一种是通过TermVectorsResponse的toXContent方法直接生成XContentBuilder，这种方法可以直接获取和上面通过DSL查询一样的json结果；第二种是通过Fields的iterator遍历fields，获取TermsEnum，熟悉lucene的同学应会更熟悉第二种方法。