es

编程语言 编程语言
Created At :
Views 👀 :
  1. 常用查询模式
  2. elasticsearch 安装
    1. 选择一个目录放 elasticsearch 项目
    2. 下载 elasticsearch
    3. 配置
    4. 启动
    5. 优化
  3. Kibana 安装
    1. 配置
    2. 启动
    3. kibana 标题修改
  4. 横向扩容
  5. 缩减节点
  6. 集群配置样例
  7. 启动监控
  8. reindex
    1. 迁移索引
  9. network.host 可以配置特殊值，并且可以配置多个
  10. ES devtool 使用请求

常用查询模式

  1. 7.0+ 精确数量
1
"track_total_hits":true,
  1. 脚本查询
1
2
3
4
5
6
7
8
9
10
11
12
13
14
GET /content_summary/_search
{
  "query": {
    "bool": {
      "must": {
        "script": {
          "script": {
            "source": "doc['item_id'].value == doc['item_gid'].value && doc['gid_seq'].value != 1"
          }
        }
      }
    }
  }
}
  1. insert
1
2
3
4
5
6
PUT twitter/_doc/1
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}
  1. update

https://www.elastic.co/guide/en/elasticsearch/reference/7.5/docs-update.html

1
2
3
4
5
6
7
POST /starhalo_video_group/_update/v16qt0hbc0
{
  "doc" : {
    "group" : "xxxxx",
    "group_seq":1
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
POST /starhalo_video_group/_update_by_query?conflicts=proceed
{
"query": {
    "query_string": {
      "query": "group:sssss AND NOT item_id:sssss"
    }
  },
  "script": {
    "source": "ctx._source.group=ctx._source.item_id;ctx._source.group_seq=1;",
    "lang": "painless"
  }
}

批量更新 https://elasticsearch.cn/question/8085

1
2
3
4
5
6
7
8
9
10
11
12
from elasticsearch import Elasticsearch
import pandas as pd

es = Elasticsearch()

doc = [
    {'update':{'_index':'ecommerce','_id':'11'}},
    {'doc':{'price':1314}},
    {'update': {'_index': 'ecommerce', '_id': '1'}},
    {'doc': {'price': 7758}},
]
result = es.bulk(body=doc, index="ecommerce")
  1. 脚本实现增删改查
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

## 修改字段值

POST test/_doc/1/_update
{
    "script" : {
        "source": "ctx._source.counter += params.count",
        "lang": "painless",
        "params" : {
            "count" : 4
        }
    }
}

## 删除值的某个元素

POST test/_doc/1/_update
{
    "script" : {
        "source": "if (ctx._source.tags.contains(params.tag)) { ctx._source.tags.remove(ctx._source.tags.indexOf(params.tag)) }",
        "lang": "painless",
        "params" : {
            "tag" : "blue"
        }
    }
}

## 添加字段

POST test/_doc/1/_update
{
    "script" : "ctx._source.new_field = 'value_of_new_field'"
}

## 删除字段

POST test/_doc/1/_update
{
    "script" : "ctx._source.remove('new_field')"
}

POST /test/_update_by_query?conflicts=proceed
{
  "query": {
    "query_string": {
      "query": "_id:abc"
    }
  },
  "script" : "ctx._source.remove('ctime','a')"
}

elasticsearch 安装

选择一个目录放 elasticsearch 项目

这里放在～/pkgs/ 目录下面
cd
mkdir pkgs
cd pkgs

下载 elasticsearch

方式一：共享目录：smb://192.168.1.100/soft/packages/elasticsearch-7.4.0-linux-x86_64.tar.gz
方式二：直接从官网下载 https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz

解压
tar  xfv elasticsearch-7.4.0-linux-x86_64.tar.gz

配置

修改系统参数
sudo sysctl -w vm.max_map_count=262144

或者：

  1. 编辑 /etc/sysctl.conf
  2. 添加 vm.max_map_count=262144
    1. 或者：echo “vm.max_map_count=262144” >> /etc/sysctl.conf
  3. 执行 sysctl -p
  4. more /proc/sys/vm/max_map_count 观察效果

sudo vim /etc/security/limits.conf
添加如下内容：

  • soft memlock unlimited
  • hard memlock unlimited

备注：* 代表 Linux 所有用户名称

修改配置文件
vim config/elasticsearch.yml

cluster.name: es_cluster           #集群名称，同一个集群的机器，此名称必须相同
node.name: node-1                   #节点名称，同一个集群的节点名称不能相同
network.host: 192.168.8.108     #节点 host
http.port: 9200                          #节点端口，不配的话默认就是 9200
transport.tcp.port: 9300            #节点 tcp 端口，不配的话默认是 9300
discovery.seed_hosts: [“172.21.1.44”, “172.21.1.45”, “172.21.1.46”]  #集群初始化 host 列表，可以将集群其他机器的 ip 写在列表里，作为节点启动时的初始化集群列表，此列表中的 ip 必须和本机在同一集群中
cluster.initial_master_nodes: [“node-1”, “node-2”]   #集群启动时，可用于选举作为主节点的机器，可以写 ip 也可以写节点名称

vim config/jvm.options
修改 jvm 内存，物理内存一半，不能超过 32G
-Xms16g
-Xmx16g

修改 gc 为 g1，
-XX:+UseG1GC

其他节点的配置类似

启动

./bin/elasticsearch， 后台运行在命令后面加 -d 参数
./bin/elasticsearch -d

优化

  1. 调整磁盘同步速率
1
2
3
4
5
6
PUT /_cluster/settings
{
  "transient": {
    "indices.recovery.max_bytes_per_sec" : "50mb"
  }
}
  1. 修改 refresh 周期（默认一秒），有实时要求的集群不设置
1
2
PUT /picture_online/_settings
{ "refresh_interval": "60s" }
  1. 增加慢查询记录
1
2
3
4
5
6
7
8
9
10
11
PUT _settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.query.debug": "1s",
  "index.search.slowlog.threshold.query.trace": "400ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "800ms",
  "index.search.slowlog.threshold.fetch.debug": "500ms",
  "index.search.slowlog.threshold.fetch.trace": "200ms"
}

Kibana 安装

kibana 与 elasticsearch 是配套的，必须安装同一版本的 kibana, 可以从官网找到需要的版本
下载地址：
https://artifacts.elastic.co/downloads/kibana/kibana-7.4.2-linux-x86_64.tar.gz

下载后解压，

配置

打开 config 下的 kibana.yml  配置文件
server.port: 5601    #默认端口是 5601
server.host: “192.168.8.108”    #修改 server.host 为本机 ip
server.name: “yourkibana”        #服务名称
elasticsearch.hosts: [“http://192.168.8.108:9200"]    #集群中 es 的 host, 一般写主节点的 ip

启动

nohup /bin/kibana &   后台启动

kibana 标题修改

6.5.4 版本修改：./src/ui/ui_render/views/chrome.pug
7.0 以上版本修改： ./src/legacy/ui/ui_render/views/chrome.pug

1
title Sh-Search

横向扩容

sh-search 计划从 3 台机器扩容到 9 台机器
操作流程如下

  1. 修改配置，保持集群名相同，修改：

node.name: node-4
network.host: 172.21.0.108

  1. 重启，观察集群可以自动发现新增节点，观察节点正常进入服务状态
  2. 逐台添加
  3. 完成后，在集群的 setting 中修改最小 master 节点数为 N/2+1
1
2
3
4
5
6
PUT /_cluster/settings
{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 5
    }
}

缩减节点

  1. 先在集群中排除此节点
1
2
3
4
5
6
PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "127.1.2.3"
  }
}
  1. 停实例

集群配置样例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cluster.name: vdm-content #集群名称

node.name: node-3 #节点名称

path.data: /home/webserver/data/es/  #数据目录

path.logs: /home/webserver/logs/logger/es/ #日志目录

network.host: 172.21.0.11 #内网地址

http.port: 9200 #端口，统一使用9200

discovery.seed_hosts: ["172.21.0.12", "172.21.0.13", "172.21.0.11"] #实例节点

cluster.initial_master_nodes: ["node-1", "node-2"] #master候选节点

启动监控

只需要开启如下配置：(7.5.0)

1
xpack.monitoring.collection.enabled: true

reindex

迁移索引

https://www.jianshu.com/p/afae616bdef5

https://blog.csdn.net/lpp_dd/article/details/78554839

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# 白名单
reindex.remote.whitelist: ["ip:9200","ip2:9200"]
or
reindex.remote.whitelist: "ip:9200,ip2:9200"

# 设置index.number_of_replicas来禁用副本：0。
PUT /my_logs/_settings
{
    "number_of_replicas": 1
}

# -1 或者足够大
PUT /my_logs/_settings
{ "refresh_interval": -1 }
PUT /my_logs
{
  "settings": {
    "refresh_interval": "30s"
  }
}

POST _reindex?wait_for_completion=false&slices=5
POST _reindex
{
  "source": {
    "remote": {
      "host": "http://10.10.10.102:9200",
      "socket_timeout": "30s",
      "connect_timeout": "30s"
    },
    "index": "voice2017-11",
    "size": 1000,
    "max_docs":1000,
    "query":{
    }
  },
  "dest": {
    "index": "voice2017-11"
  }
}

network.host 可以配置特殊值，并且可以配置多个

1
network.host: _local_, _site_

ES devtool 使用请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309

GET _search
{
  "query": {
    "match_all": {}
  }
}

GET /starhalo_moment_group/_search
{
  "query": {
    "query_string": {
      "query": "item_id:(v0zcp33bxx)"
    }
  },
  "_source": ["group","group_seq"],
  "size":1000
}

POST /starhalo_picture_group/_doc/p132d003vm
{
  "item_id" : "p132d003vm",
    "ctime" : 1568712928,
    "post_date" : "2009-11-15T14:12:12",
    "group_seq" : 1
}

GET /starhalo_picture_group_v2/_settings
PUT /starhalo_picture_group_v2/_settings
{
  "refresh_interval":"60s",
  "number_of_replicas":1
}

GET _tasks?detailed=true&actions=*recovery

POST /starhalo_video_group/_update/v16qt0hbc0
{
  "doc" : {
    "group" : "v0nsd0pr43",
    "group_seq":100
  }
}

POST /starhalo_video_group/_updatessss_by_query?conflicts=proceed
{
"query": {
    "query_string": {
      "query": "group:v0nsd0pr43 AND NOT item_id:v0nsd0pr43"
    }
  },
  "script": {
    "source": "ctx._source.group=ctx._source.item_id;ctx._source.group_seq=1;",
    "lang": "painless"
  }
}

PUT /_template/image_v2
{
  "order": 1,
  "index_patterns": [
    "*_image_v2"
  ],
  "settings": {
    "index": {
      "number_of_shards": "4",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
      "dynamic": "true",
      "dynamic_templates": [
        {
          "hash": {
            "match":   "*_ha",
            "mapping": {
              "type": "text"
            }
          }
        },
        {
          "histogram": {
            "match":   "*_hi",
            "mapping": {
              "type": "binary",
              "store":true
            }
          }
        }
      ],
      "properties": {
        "item_id": {
          "type": "keyword"
        },
        "image_url": {
          "type": "keyword"
        },
        "image_file": {
          "type": "keyword"
        }
      }
    }
}

PUT /_template/group_v2
{
  "order": 1,
  "index_patterns": [
    "*_group_v2"
  ],
  "settings": {
    "index": {
      "number_of_shards": "2",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
      "dynamic": "true",
      "properties": {
        "item_id": {
          "type": "keyword"
        },
        "ctime": {
          "type": "long"
        },
        "md5": {
          "type": "keyword"
        },
        "same":{
          "type":"keyword"
        },
        "similary":{
          "type":"long"
        },
        "group":{
          "type":"keyword"
        },
        "group_similary":{
          "type":"long"
        },
        "group_seq":{
          "type":"long"
        }
      }
    }
}

PUT /starhalo_picture_image_v1
PUT /starhalo_picture_image_v1/_alias/starhalo_picture_image
PUT /starhalo_picture_group_v1
PUT /starhalo_picture_group_v1/_alias/starhalo_picture_group

GET /starhalo_picture_image/_mapping

GET /starhalo_picture_image/_search
{
  "track_total_hits": true,
  "query": {
    "query_string": {
      "query": "_exists_:phash"
    }
  }
}

#DELETE /starhalo_picture_simple_image_v2

GET /_cat/aliases

POST _reindex?wait_for_completion=true
{
  "source": {
    "remote": {
      "host": "http://192.168.139.76:9200"
    },
    "index": "starhalo_picture_group",
    "size": 1000,
    "query": {
      "query_string": {
        "query": "ctime:>1576814400"
      }
    }
  },
  "dest": {
    "index": "starhalo_picture_group",
    "op_type":"create"
  },
  "conflicts":"proceed"
}

POST _reindex?wait_for_completion=true
{
  "source": {
    "index": "starhalo_picture_group_v1",
    "query": {
      "query_string": {
        "query": "ctime:>1576814400"
      }
    },
    "size": 1000
  },
  "dest": {
    "index": "starhalo_picture_group_v2",
    "op_type":"create"
  },
  "conflicts":"proceed"
}

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "starhalo_moment_group_v2",
        "alias": "starhalo_moment_group"
      }
    },
    {
      "remove": {
        "index": "starhalo_moment_group_v1",
        "alias": "starhalo_moment_group"
      }
    }
  ]
}
#查看进度
GET _tasks?detailed=true&actions=*reindex
GET _tasks?detailed=true&actions=*recovery
GET _tasks?detailed=true&actions=*byquery

POST /starhalo_video_simple_image_v2/_update_by_query?conflicts=proceed&wait_for_completion=false
{
  "query": {
    "query_string": {
      "query": "_exists_:eh_hi AND ctime:<1570636800"
    }
  },
  "script" : "for(def key:['eh','ad','ce']){ for(def suf:['_hi','_ha']) {ctx._source.remove(key+suf) } }"
}

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "192.168.52.192,192.168.52.193"
  }
}

GET _cluster/settings/
PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.enable" : "all",
    "cluster.routing.rebalance.enable" : "all"
  }
}

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : null
  }
}

PUT _cluster/settings
{
  "transient": {
    "indices.recovery.max_bytes_per_sec":"512mb",
    "indices.recovery.max_concurrent_file_chunks":5
  }
}

POST /_cluster/reroute
{
    "commands" : [
        {
            "move" : {
                "index" : "starhalo_moment_simple_image_v2",
                "shard" : 0,
                "from_node" : "in-ds-001",
                "to_node" : "node-1"
            }
        },
        {
            "move" : {
                "index" : "starhalo_video_image_v3",
                "shard" : 3,
                "from_node" : "in-ds-001",
                "to_node" : "node-2"
            }
        }
    ]
}


PUT /starhalo_*_image_*/_settings
{
  "refresh_interval":"60s",
  "number_of_replicas":0
}
PUT /starhalo_*_group_*/_settings
{
  "refresh_interval":"10s",
  "number_of_replicas":0
}

PUT /.*/_settings
{
  "number_of_replicas":0
}

转载请注明来源，欢迎对文章中的引用来源进行考证，欢迎指出任何有错误或不够清晰的表达。可以在下面评论区评论，也可以邮件至 jaytp@qq.com

💰

©2016-2020 Yelog

Built with Hexo and 3-hexo theme

×

Help us with donation