ElasticSearch介绍
ElasticSearch是一个
基于Lucene的搜索服务器;
它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口;
Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是当前流行的企业级搜索引擎(阿里巴巴、Google、京东等都在使用);
设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便;
我们建立一个网站或应用程序，并要添加搜索功能，但是想要完成搜索工作的创建是非常困难的。我们希望搜索解决方案要运行速度快，我们希望能有一个零配置和一个
完全免费的搜索模式，我们希望能够简单地使用JSON通过HTTP来索引数据，我们希望我们的搜索服务器始终可用，我们希望能够从一台开始并扩展到数百台，我们要实时搜索，我们要简单的多租户，我们希望建立一个云的解决方案。因此我们利用Elasticsearch来解决所有这些问题以及可能出现的更多其它问题;
## 安装环境 * CenterOS7 * JDK1.8 * ElasticSearch-5.6.4 (https://www.elastic.co/downloads/past-releases) * elasticsearch-head (ElasticSearch插件 https://github.com/mobz/elasticsearch-head) * Node.js-v8.9.0
## 安装步骤 1 . 上传下载好的软件,如下 
2 . 安装和配置好JDK和Node.js
自行Google或百度
3 . 解压好ElasticSearch
## 配置ElasticSearch和Linux * 配置ElasticSearch [root@localhost local]# vim ./elasticsearch-5.6.4/config/elasticsearch.yml 
-
配置Linux
注意：ElasticSearch是不允许root用户启动的，所以需要使用其他用户来进行启动操作，以下是新的用户组和用户的创建：[root@localhost bin]$ useradd tandi # 新增一个用户tandi [root@localhost bin]$ passwd tandi # 修改用户tandi的密码 [root@localhost bin]$ groupadd elastic # 新增一个用户组elastic [root@localhost bin]$ usermod -a -G elastic tandi # 将用户tandi添加到组elastic中 [root@localhost bin]$ chown -R tandi:elastic elasticsearch-5.6.4 # 将elasticsearch-6.1.1/权限给elastic组tandi这个用户 [root@localhost bin]$ su tandi # 切换到tandi用户 [tandi@localhost bin]$ ./elasticsearch -d # 后台运行ElasticSearch 可以使用命令jps来查看是否运行成功，成功如下： [tandi@localhost bin]$ jps 32268 Elasticsearch 32285 Jps
## 启动ElasticSearch期间可能会出现的问题解决方法 问题：[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] ``` 解决方法： [root@localhost local]# vim /etc/security/limits.conf
在文件后追加以下内容：
- hard nofile 65536
- soft nofile 65536
如下：
<br>
问题：[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决方法：
[root@localhost local]# vim /etc/sysctl.conf
在文件后追加以下内容：
vm.max_map_count=655360
如下：
>其余问题参考以下文章和自行Google或百度
>elasticsearch-5.1.1 安装的问题 http://blog.csdn.net/xiegh2014/article/details/53771086
>CentOS7上elasticsearch5.5启动报错 http://blog.java1234.com/blog/articles/342.html
<br>
## 浏览器访问ElasticSearch
<br>
## 安装elasticsearch-head插件
该插件的作用是让ElasticSearch在浏览器的操作界面更加友好
1. 首选解压elasticsearch-head
[root@localhost myfile]# unzip elasticsearch-head-master.zip
如下：
2. 进入elasticsearch-head-master使用nmp命令进行安装操作
[root@localhost elasticsearch-head-master]# npm install
如下：
提示：如果npm安装依赖慢，建议将npm设置为国内镜像，又或者直接在主机中安装该插件，如下：
3. 浏览器访问head插件
4. 配置ElasticSearch让其和head插件关联起来
[root@localhost config]# vim elasticsearch.yml
在文本最后追加以下内容,并重启ElasticSearch：
http.cors.enabled: true
http.cors.allow-origin: "*"
如下：
5. 浏览器中使用head插件连接ElasticSearch
<br>
## 配置ElasticSearch集群
1. 以当前ElasticSearch作为源复制三分到elasticsearch-cluster文件夹中，如下：
`切记:如果使用之前已有数据的ElasticSearch复制，复制完后记得删除各个ElasticSearch中的data目录，否者集群不成功！！！（问题：http://blog.csdn.net/qq_24879495/article/details/77718032）`
2. master配置不用动，其余两个从节点配置大体相同（slave1配置如下）：
3. 问题
启动过程中可能会出现问题：Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory
原因：因为三个elasticsearch在同一台虚拟机中运行，这是系统内存不足所致，elasticsearch启动内容默认为2G
解决：
[root@localhost elasticsearch-cluster]# vim elasticsearch-master/config/jvm.options
[root@localhost elasticsearch-cluster]# vim elasticsearch-slave1/config/jvm.options
[root@localhost elasticsearch-cluster]# vim elasticsearch-slave2/config/jvm.options
<br>
## ElasticSearch中对索引的基本操作
### 对索引的基本理解
* 索引：可以理解为书的目录（目录=索引）
* 类型：索引的类型，可以理解为目录中的分类标题
* 文档：类型下的具体内容
结合下图理解（个人理解）：
### 创建索引
参考官方文档，如下：(https://www.elastic.co/guide/cn/elasticsearch/guide/current/index-templates.html)
自定义例子（postman）：
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1
},
"mappings": {
"man": {
"properties": {
"name": {
"type": "text"
},
"country": {
"type": "keyword"
},
"age": {
"type": "integer"
},
"date": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}
}
},
"woman": {
},
"superman": {
}
}
}
结果：
注意：以上设置都不是必须自己设置的，现在尝试创建一个只指定索引名称但不写任何结构的索引，看看是什么情况
请求URL： 192.168.229.100：9200/peoplex
以下是自动生成的索引结构
{
"state": "open",
"settings":
{
"index":
{
"creation_date": "1514247896163",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "tThX8Af3TyWFjA0FJjIsnw",
"version":
{
"created": "5060499"
},
"provided_name": "peoplex"
}
},
"mappings":
{},
"aliases": [],
"primary_terms":
{
"0": 1,
"1": 1,
"2": 1,
"3": 1,
"4": 1
},
"in_sync_allocations":
{
"0": [
"z7DpqK6OROa9XbYNgQPOWQ",
"54ZWMzLHQoyMPfxh1nE69A"
],
"1": [
"LCIDY_G5R1SLhi2mOWUOWw",
"uzBW-bxfSiSgJVfeqkI49A"
],
"2": [
"J1m9dQ2NQvykqr12Fg5EYw",
"8anEJ_TOSyGMoeJU0p_C5w"
],
"3": [
"SKhsMMwjR4iQxXvrhsvyGg",
"vBibauuNSIe-k0dqEQuqLQ"
],
"4": [
"AhfRDd6tTDKr20kU02WnoA",
"_eFgNTl3SuecNu79ZbVLSw"
]
}
}
### 增删改查
和ElasticSearch交互：https://www.elastic.co/guide/cn/elasticsearch/guide/current/_talking_to_elasticsearch.html
#### 增
指定id：
不指定id：
#### 删
删除文档：
删除索引：
#### 改
删除文档：
/////////////////1
/////////////////2
/////////////////3
#### 查
1. 获取指定id文档
2. 查询全部
3. 字段查找
4. 分组查找
5. 统计查询
6. 精确查询
7. 多字段查找
8. 范围查找
9. 过滤查找
10. 表达式查找
#### 数据返回格式参照
//URL get请求返回数据格式 （192.168.229.100:9200/people/man/AWCDzUmToHzKzvfCjwMr）
{
"_index": "people",
"_type": "man",
"_id": "AWCDzUmToHzKzvfCjwMr",
"_version": 1,
"found": true,
"_source": {
"name": "李四",
"age": 23,
"country": "加拿大",
"date": "2000-12-01"
}
}
//post请求返回数据格式(以下为分组查找的响应格式)
{
"took": 25,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 12,
"max_score": 1,
"hits": [
{
"_index": "people",
"_type": "man",
"_id": "AWCD6XPXoHzKzvfCjwMx",
"_score": 1,
"_source": {
"name": "赵四",
"age": 19,
"country": "泰国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCEAQFYoHzKzvfCjwMz",
"_score": 1,
"_source": {
"name": "王中天",
"age": 19,
"country": "中国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCDzOc0oHzKzvfCjwMq",
"_score": 1,
"_source": {
"name": "张三",
"age": 21,
"country": "美国",
"date": "1997-05-05"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCDzUmToHzKzvfCjwMr",
"_score": 1,
"_source": {
"name": "李四",
"age": 23,
"country": "加拿大",
"date": "2000-12-01"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCIbziXASUp1WxfELkt",
"_score": 1,
"_source": {
"name": "桃白白",
"age": 19,
"country": "泰国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCIb6WVASUp1WxfELku",
"_score": 1,
"_source": {
"name": "谭迪",
"age": 0,
"country": "泰国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCDzjLYoHzKzvfCjwMs",
"_score": 1,
"_source": {
"name": "王五",
"age": 18,
"country": "泰国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCD1meEoHzKzvfCjwMv",
"_score": 1,
"_source": {
"name": "王五2",
"age": 18,
"country": "泰国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCD1pHToHzKzvfCjwMw",
"_score": 1,
"_source": {
"name": "赵四",
"age": 18,
"country": "泰国",
"date": "1992-08-17"
}
},
{
"_index": "people",
"_type": "man",
"_id": "AWCIBMplASUp1WxfELkj",
"_score": 1,
"_source": {
"name": "谭迪",
"age": 88,
"country": "中国",
"date": null
}
}
]
},
"aggregations": {
"以date分组": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": 714009600000,
"key_as_string": "1992-08-17 00:00:00",
"doc_count": 8
},
{
"key": 862790400000,
"key_as_string": "1997-05-05 00:00:00",
"doc_count": 1
},
{
"key": 975628800000,
"key_as_string": "2000-12-01 00:00:00",
"doc_count": 1
}
]
},
"以age分组": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": 19,
"doc_count": 4
},
{
"key": 18,
"doc_count": 3
},
{
"key": 21,
"doc_count": 2
},
{
"key": 0,
"doc_count": 1
},
{
"key": 23,
"doc_count": 1
},
{
"key": 88,
"doc_count": 1
}
]
}
}
}
>以上都是些较为简单的例子，推荐参考以下资料来学习
>ElasticSearch权威指南：https://www.elastic.co/guide/cn/elasticsearch/guide/current/ structured-search.html
>ElasticSearch 常用的查询过滤语句：https://www.cnblogs.com/ghj1976/p/5293250.html
<br>
## 使用java client api基本使用
快速入门案例：spring boot + maven + elasticsearch
项目目录结构：
以下为主要代码和配置：
pom.xml
<groupId>com.td</groupId>
<artifactId>estest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>estest</name>
<description></description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.9.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<!-- 属性 -->
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<!-- 依赖 -->
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- elasticSearch -->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>6.1.1</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.9.1</version>
</dependency>
<!-- lang3 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
pojo：
public class People {
//{"date":"1996-05-05","country":"中国","name":"谭迪","age":21}
private String name;
private int age;
private String country;
private String date;
public People() {
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
public String getCountry() {
return country;
}
public void setCountry(String country) {
this.country = country;
}
public String getDate() {
return date;
}
public void setDate(String date)throws Exception{
this.date = date;
}
@Override
public String toString() {
return "People{" +
"name=\'" + name + \'\\\'\' +
", age=" + age +
", country=\'" + country + \'\\\'\' +
", date=" + date +
\'}\';
}
}
configuration（spring4-javaConfig）：
@Configuration
public class ESJavaConfig {
/**
* 返回ElasticSearch 客户端api对象
* @return
* @throws Exception
*/
@Bean
public TransportClient client()throws Exception {
Settings.Builder builder = Settings.builder();
builder.put("cluster.name", "ElasticSearchCluster"); //集群名称
builder.put("client.transport.sniff", true); //开启嗅探（简单来说就是你不用配置集群中所有的es的ip，开启嗅探后将会自动查找）
builder.put("client.transport.ignore_cluster_name", false); //是否忽略集群名称校验
builder.put("client.transport.ping_timeout", "5s"); //超时时间
builder.put("client.transport.nodes_sampler_interval", "5s"); //取样时间（刷新）
Settings settings = builder.build();
TransportClient client = new PreBuiltTransportClient(settings)
.addTransportAddress(new TransportAddress(InetAddress.getByName("192.168.229.100"), 9300));
return client;
}
}
controller：
@RestController
public class ESController {
@Autowired
TransportClient client;
/**
* URL id查询
* @param id
* @return
*/
@GetMapping("/get/people/{id}")
public ResponseEntity findPeopleById(@PathVariable("id") String id){
//使用客户单向es发出查找请求
GetRequestBuilder getRequestBuilder = client.prepareGet("people", "man", id);
//获取es响应结果对象
GetResponse response = getRequestBuilder.get();
//向浏览器返回json格式的结果
return new ResponseEntity(response.getSource(), HttpStatus.OK);
}
/**
* 接受表单形式参数来创建文档
* @param people
* @return
* @throws Exception
*/
@PostMapping("/post/people")
public ResponseEntity addPeople(People people) throws Exception {
//创建json构建对象XContentBuilder，并构建json
XContentBuilder xContentBuilder = XContentFactory.jsonBuilder()
.startObject()
.field("name", people.getName())
.field("age", people.getAge())
.field("country", people.getCountry())
.field("date", people.getDate())
.endObject();
//使用客户端想es发出创建文档请求
IndexResponse indexResponse = client.prepareIndex("people", "man")
.setSource(xContentBuilder)
.get();
//向浏览器返回json格式的结果
return new ResponseEntity(indexResponse.getId(), HttpStatus.OK);
}
/**
* 接受json串形式参数创建对象
* @param json
* @return
* @throws Exception
*/
@PostMapping ("/post/json2people")
public ResponseEntity addJson2People(@RequestBody String json) throws Exception {
//将json串转为pojo
People people = JsonUtils.jsonToPojo(json, People.class);
//创建json构建对象XContentBuilder，并构建json
XContentBuilder xContentBuilder = XContentFactory.jsonBuilder()
.startObject()
.field("name", people.getName())
.field("age", people.getAge())
.field("country", people.getCountry())
.field("date", people.getDate()+"")
.endObject();
//使用客户端想es发出创建文档请求
IndexResponse indexResponse = client.prepareIndex("people", "man")
.setSource(xContentBuilder) //将构建好的文档json对象发送给es
.get(); //获取es响应结果对象IndexResponse
//向浏览器返回json格式的结果
return new ResponseEntity(indexResponse.getId(), HttpStatus.OK);
}
/**
* 根据id删除文档
* @param id
* @return
*/
@DeleteMapping("/delete/people/{id}")
public ResponseEntity deletePeopleById(@PathVariable("id") String id){
//使用客户端向es发出删除请求，并返回响应删除请求的结果对象DeleteRequestBuilder
DeleteRequestBuilder deleteRequestBuilder = client.prepareDelete("people", "man", id);
//从DeleteRequestBuilder中获取请求结果
DeleteResponse deleteResponse = deleteRequestBuilder.get();
//向浏览器返回json格式的结果
return new ResponseEntity(deleteResponse.getId(), HttpStatus.OK);
}
/**
* 接受表单形式参数来修改指定id的文档
* @param id
* @param json
* @return
* @throws Exception
*/
@PutMapping("/update/people/{id}")
public ResponseEntity updatePeopleById(@PathVariable("id") String id, @RequestBody String json) throws Exception{
//将json串转为pojo对象
People people = JsonUtils.jsonToPojo(json, People.class);
//创建更新请求对象
UpdateRequest updateRequest = new UpdateRequest("people", "man", id);
//创建构建json对象的XContentBuilder
XContentBuilder xContentBuilder = XContentFactory.jsonBuilder();
xContentBuilder.startObject(); //构建开始
if( people.getName() != null ){
xContentBuilder.field("name", people.getName());
}
if ( people.getCountry() != null ){
xContentBuilder.field("country", people.getCountry());
}
if ( people.getDate() != null ){
xContentBuilder.field("date", people.getDate());
}
if ( people.getAge() != 0 ){
xContentBuilder.field("age", people.getAge());
}
xContentBuilder.endObject(); //构建结束
//将构建好的json对象添加到更新请求对象中
UpdateRequest request = updateRequest.doc(xContentBuilder);
//使用客户端想es发出更新请求，并返回请求结果
UpdateResponse updateResponse = client.update(request).get();
//向浏览器返回json格式的结果
return new ResponseEntity(updateResponse.getId(), HttpStatus.OK);
}
}
附加一段复合查询代码(不属于上面项目)
@PostMapping("query/book/novel")
public ResponseEntity query( @RequestParam(value = "gt_word_count", defaultValue = "0") int gtWordCount
, @RequestParam(value = "author", required =false) String author
, @RequestParam(value = "title", required =false) String title
, @RequestParam(value = "lt_word_count", required = false) Integer ltWordCount) {
//使用QueryBuilders构建 bool查询构建对象 并返回BoolQueryBuilder
BoolQueryBuilder boolBuilder = QueryBuilders.boolQuery();
if (author != null) {
//must相当于sql中的and
boolBuilder.must( QueryBuilders.matchQuery("author", author ) );
}
if (title != null) {
boolBuilder.must( QueryBuilders.matchQuery("title", title ) );
}
if (bookBean.getTitle() != null) {
builder.must( QueryBuilders.matchQuery("title", bookBean.getTitle()) );
}
//使用QueryBuilders构建 range范围查询构建对象 并返回RangeQueryBuilder
RangeQueryBuilder rangeQuery = QueryBuilders.rangeQuery("word_count").from(gtWordCount);
if (ltWordCount != null) {
rangeQuery.to(ltWordCount);
}
//将 range范围查询构建对象 作为过滤条件添加进 bool查询构建对象
boolBuilder.filter(rangeQuery);
//使用客户端向es发出查找请求，并返回 查找请求构建对象
SearchRequestBuilder builder = this.client.prepareSearch("book")
.setTypes("novel") //查找的是 novel 类型
.setSearchType(SearchType.QUERY_THEN_FETCH) //设置检索类型（参考文章：http://www.linuxidc.com/Linux/2015-02/114248.htm）
.setQuery(boolBuilder) //设入 查询构建对象（内包含着整个查询要求）
.setFrom(0) //从索引0开始查找
.setSize(10); //查找到索引10
//打印es响应的 查找请求构建对象
log.info(String.valueOf(builder));
//从响应中过去请求响应结果
SearchResponse response = builder.get();
//创建一个存放查找结果的容器
List<Map<String,Object>> result = new ArrayList<>();
//遍历查找的的hits集合【java8 lambda表达式】
response.getHits().forEach((s)->result.add(s.getSource()));
//将查找结果返回给浏览器
return new ResponseEntity(result, HttpStatus.OK);
}