分布式全文搜索引擎——Elasticsearch
Posted vettel0329
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了分布式全文搜索引擎——Elasticsearch相关的知识，希望对你有一定的参考价值。
1.安装Elasticsearch
a.下载：从官网下载 Elasticsearch，地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch
b.启动：解压后，在 bin 目录打开 elasticsearch.bat 启动服务
c.访问：在浏览器中访问 http://127.0.0.1:9200/ 检查是否安装成功
注意：端口9300：Java程序访问的端口；端口9200：浏览器访问的端口
Elasticsearch 的版本需要跟后面的 Spring-Data-Elasticsearch 的版本相符，不然会不兼容（Spring 5.1.1 对应 Elasticsearch 6.8.0）
2.安装Head管理Elasticsearch插件
a.安装nodejs：在nodejs官网下载msi，地址：https://nodejs.org/en/download/，安装后在CMD命令行输入 node -v 查看是否安装成功
b.安装grunt：在CMD命令行输入 npm install -g grunt-cli 安装grunt，安装后输入 grunt -version 查看是否安装成功
c.下载Head：在GitHub下载并解压 elasticsearch-head-master，地址：https://github.com/mobz/elasticsearch-head
d.修改Head配置：修改Head根目录下Gruntfile.js，添加：hostname:‘*‘,
e.修改Elasticsearch配置：修改 Elasticsearch 的 config 目录下的 elasticsearch.yml 文件，添加如下配置。并重启
network.host: 127.0.0.1 # 解决elasticsearch-head 集群健康值: 未连接问题 http.cors.enabled: true http.cors.allow-origin: "*"
f.安装Head：在 elasticsearch-head-master 根目录下执行命令 npm install 进行安装
g.启动Head：同样在 elasticsearch-head-master 根目录下执行命令 grunt server（或者npm run start）
h.访问Head：在浏览器中访问 http://localhost:9100/ 即可进入管理界面
3.安装IK中文分词器
a.下载：在GitHub上下载对应版本的IK分词器zip包，地址：https://github.com/medcl/elasticsearch-analysis-ik/releases
b.注意：Elasticsearch和IK分词器必须版本统一
c.解压安装：解压到 elasticsearch 的 plugins 目录下，并改名为 ik。并重启 elasticsearch
4.使用 Spring-Data-Elasticsearch 操作 Elasticsearch
a.导入pom依赖
<properties> ...... <!-- spring --> <spring.version>5.1.1.RELEASE</spring.version> </properties> <dependencies> <!-- elasticsearch --> <dependency> <groupId>org.springframework.data</groupId> <artifactId>spring-data-elasticsearch</artifactId> <version>3.1.1.RELEASE</version> </dependency> <!-- spring-5.X --> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-core</artifactId> <version>$spring.version</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-webmvc</artifactId> <version>$spring.version</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-oxm</artifactId> <version>$spring.version</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-tx</artifactId> <version>$spring.version</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-aop</artifactId> <version>$spring.version</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-context-support</artifactId> <version>$spring.version</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-test</artifactId> <version>$spring.version</version> </dependency> <!-- AOP-AspectJ spring-aop依赖 --> <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjrt</artifactId> <version>1.8.6</version> </dependency> <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjweaver</artifactId> <version>1.8.6</version> </dependency> <!-- jackson-json spring-mvc依赖 --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.9.4</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-core</artifactId> <version>2.9.4</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-annotations</artifactId> <version>2.9.4</version> </dependency> <!-- fastjson --> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.47</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> <scope>test</scope> </dependency> </dependencies>
b.创建BaseEntity基础实体类
public class BaseEntity @Id private String id; public String getId() return id; public void setId(String id) this.id = id;
c.创建Article和Author实体类
//index相当于数据库，type相当于表 @Document(indexName = "elasticsearch", type = "article") public class Article extends BaseEntity private String title; private String content; //内嵌对象 @Field(type = FieldType.Nested) private List<Author> authors; public String getTitle() return title; public void setTitle(String title) this.title = title; public List<Author> getAuthors() return authors; public void setAuthors(List<Author> authors) this.authors = authors; public String getContent() return content; public void setContent(String content) this.content = content; @Override public String toString() return "Article: id[" + this.getId() + "], title[" + title + "], content[" + content + "], authors[" + authors + "]";
public class Author private String name; private int year; public String getName() return name; public void setName(String name) this.name = name; public int getYear() return year; public void setYear(int year) this.year = year; @Override public String toString() return "Author: name[" + name + "], year[" + year + "]";
d.创建BaseDao基础DAO
public class BaseDao<T extends BaseEntity> @Resource(name="elasticsearchTemplate") protected ElasticsearchTemplate esTemplate; protected Class<T> clazz; @PostConstruct private void construct() clazz = (Class)((ParameterizedType)this.getClass().getGenericSuperclass()).getActualTypeArguments()[0]; /** * 保存单个对象 * @param t 对象 * @return */ public boolean save(T t) String id = t.getId(); if (id == null) id = UUID.randomUUID().toString().replaceAll("-", ""); t.setId(id); IndexQuery indexQuery = new IndexQueryBuilder().withId(id).withObject(t).build(); esTemplate.index(indexQuery); return true; /** * 保存多个 * @param list 对象集合 * @return */ public boolean save(List<T> list) List<IndexQuery> queries = new ArrayList<IndexQuery>(); for (T t : list) String id = t.getId(); if (id == null) id = UUID.randomUUID().toString().replaceAll("-", ""); t.setId(id); IndexQuery indexQuery = new IndexQueryBuilder().withId(id).withObject(t).build(); queries.add(indexQuery); esTemplate.bulkIndex(queries); return true; /** * 根据ID删除 * @param id 对象ID * @return */ public boolean deleteById(String id) esTemplate.delete(clazz, id); return true; /** * 根据多个ID删除 * @param idList 对象ID集合 * @return */ public boolean deleteByIds(List<String> idList) DeleteQuery deleteQuery = new DeleteQuery(); BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery(); if(idList != null) for (String id : idList) queryBuilder.should(QueryBuilders.matchQuery("id", id)); deleteQuery.setQuery(queryBuilder);; esTemplate.delete(deleteQuery, clazz); return true; /** * 根据过滤条件删除 * @param filter 过滤条件Map * @return */ public boolean delete(Map<String,Object> filter) DeleteQuery deleteQuery = new DeleteQuery(); BoolQueryBuilder queryBuilder = QueryBuilders. boolQuery(); if(filter != null) for (Map.Entry<String, Object> entry : filter.entrySet()) //"content.keyword" 表示不进行分词（精准查询），解决 "content" 分词后 term 查不到数据的问题 queryBuilder.must(QueryBuilders.matchQuery(entry.getKey() + ".keyword", entry.getValue())); deleteQuery.setQuery(queryBuilder); esTemplate.delete(deleteQuery, clazz); return true; /** * 根据条件查询集合 * @param filter 过滤条件Map * @param highFields 高亮字段 * @param sortField 排序字段 * @param order 正序倒序 * @return */ public List<T> queryList(Map<String, Object> filter, final List<String> highFields, String sortField, SortOrder order) NativeSearchQueryBuilder searchBuilder = new NativeSearchQueryBuilder(); //-----------------------------高亮----------------------------- List<Field> fieldList = new ArrayList<>(); if(highFields != null) for (String highField : highFields) fieldList.add(new HighlightBuilder.Field(highField).preTags("<em>").postTags("</em>").fragmentSize(250)); searchBuilder.withHighlightFields(fieldList.toArray(new Field[fieldList.size()])); //-----------------------------排序----------------------------- if (sortField != null && order != null) searchBuilder.withSort(new FieldSortBuilder(sortField + ".keyword").order(order)); //-----------------------------过滤条件----------------------------- BoolQueryBuilder queryBuilder= QueryBuilders.boolQuery(); for (Map.Entry<String, Object> entry : filter.entrySet()) queryBuilder.must(QueryBuilders.matchQuery(entry.getKey(), entry.getValue())); searchBuilder.withQuery(queryBuilder); //-----------------------------查询建立----------------------------- SearchQuery searchQuery = searchBuilder.build(); Page<T> page = null; //如果设置高亮 if (highFields != null && highFields.size() > 0) page = esTemplate.queryForPage(searchQuery, clazz, new SearchResultMapper() @Override public <T> AggregatedPage<T> mapResults(SearchResponse response,Class<T> clazz, Pageable pageable) List<T> list = new ArrayList<T>(); for (SearchHit searchHit : response.getHits()) if (response.getHits().getHits().length <= 0) return null; Map<String, Object> entityMap = searchHit.getSourceAsMap(); for (String highName : highFields) String highValue = searchHit.getHighlightFields().get(highName).fragments()[0].toString(); entityMap.put(highName, highValue); T t = JSONObject.parseObject(JSONObject.toJSONString(entityMap), clazz); list.add(t); if (list.size() > 0) return new AggregatedPageImpl<T>(list); return null; ); //如果不设置高亮 else page = esTemplate.queryForPage(searchQuery, clazz); return page.getContent(); /** * 根据条件查询分页列表 * @param filter 过滤条件Map * @param highFields 高亮字段 * @param sortField 排序字段 * @param order 正序倒序 * @param pageIndex 当前页 * @param pageSize 分页大小 * @return */ public Map<String, Object> queryPage(Map<String,Object> filter, final List<String> highFields, String sortField, SortOrder order, int pageIndex, int pageSize) NativeSearchQueryBuilder searchBuilder = new NativeSearchQueryBuilder(); //-----------------------------高亮----------------------------- List<Field> fieldList = new ArrayList<>(); if(highFields != null) for (String highField : highFields) fieldList.add(new HighlightBuilder.Field(highField).preTags("<em>").postTags("</em>").fragmentSize(250)); searchBuilder.withHighlightFields(fieldList.toArray(new Field[fieldList.size()])); //-----------------------------排序----------------------------- if (sortField != null && order != null) searchBuilder.withSort(new FieldSortBuilder(sortField + ".keyword").order(order)); //-----------------------------分页----------------------------- searchBuilder.withPageable(PageRequest.of(pageIndex, pageSize)); //-----------------------------过滤条件----------------------------- BoolQueryBuilder queryBuilder= QueryBuilders.boolQuery(); for (Map.Entry<String, Object> entry : filter.entrySet()) queryBuilder.must(QueryBuilders.matchQuery(entry.getKey(), entry.getValue())); searchBuilder.withQuery(queryBuilder); //-----------------------------查询建立----------------------------- SearchQuery searchQuery = searchBuilder.build(); Page<T> page = null; //如果设置高亮 if (highFields != null && highFields.size() > 0) page = esTemplate.queryForPage(searchQuery, clazz, new SearchResultMapper() @Override public <T> AggregatedPage<T> mapResults(SearchResponse response,Class<T> clazz, Pageable pageable) List<T> list = new ArrayList<T>(); for (SearchHit searchHit : response.getHits()) if (response.getHits().getHits().length <= 0) return null; Map<String, Object> entityMap = searchHit.getSourceAsMap(); for (String highName : highFields) String highValue = searchHit.getHighlightFields().get(highName).fragments()[0].toString(); entityMap.put(highName, highValue); T t = JSONObject.parseObject(JSONObject.toJSONString(entityMap), clazz); list.add(t); if (list.size() > 0) return new AggregatedPageImpl<T>(list); return null; ); //如果不设置高亮 else page = esTemplate.queryForPage(searchQuery, clazz); //-----------------------------查询数据总条数----------------------------- searchQuery = new NativeSearchQueryBuilder().withQuery(queryBuilder).build(); long totalCount = esTemplate.count(searchQuery, clazz); //组装结果 Map<String, Object> resultMap = new HashMap<>(); resultMap.put("pageIndex", pageIndex); resultMap.put("pageSize", pageSize); resultMap.put("data", page.getContent()); resultMap.put("totalCount", totalCount); return resultMap;
参考文档：https://blog.csdn.net/chen_2890/article/details/83757022
https://blog.csdn.net/lihuanlin93/article/details/83448967
以上是关于分布式全文搜索引擎——Elasticsearch的主要内容，如果未能解决你的问题，请参考以下文章