docker-elk集群实践

发表于 更新于 阅读次数： 阅读次数：

Uu3mWe

docker-compose安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
version: '3.0'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
      - TZ=Asia/Shanghai
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
        - ./elasticsearch/analysis/synonym.txt:/usr/share/elasticsearch/config/analysis/synonym.txt
        - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
        - ./elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
        - ./elasticsearch/config/crack/x-pack-core-7.13.0.jar:/usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.0.jar
        - ./elasticsearch/data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
      - TZ=Asia/Shanghai
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
        - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
        - ./elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
        - ./elasticsearch/config/crack/x-pack-core-7.13.0.jar:/usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.0.jar
        - ./elasticsearch/data02:/usr/share/elasticsearch/data
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
      - TZ=Asia/Shanghai
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
        - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
        - ./elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
        - ./elasticsearch/config/crack/x-pack-core-7.13.0.jar:/usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.0.jar
        - ./elasticsearch/data03:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.0
    container_name: kibana
    restart: always
    ports:
      - 5601:5601
    volumes:
        - ./kibana/config:/usr/share/kibana/config
    environment:
      I18N_LOCALE: zh-CN
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: '["https://es01:9200","https://es02:9200","https://es03:9200"]'

  ent-search:
    image: docker.elastic.co/enterprise-search/enterprise-search:7.13.0
    container_name: ent-search
    environment:
      - "JAVA_OPTS=-Xms2048m -Xmx2048m"
    volumes:
        - ./enterprise-search/config/enterprise-search.yml:/usr/share/enterprise-search/config/enterprise-search.yml
        - ./enterprise-search/config/certs:/usr/share/enterprise-search/config/certs
    ports:
    - 3002:3002

  cerebro:
    image: lmenezes/cerebro:0.9.4
    container_name: cerebro
    restart: always
    ports:
        - 8900:9000
    command:
        - -Dhosts.0.host=https://es01:9200
        - -Dplay.ws.ssl.loose.acceptAnyCertificate=true

  es-head:
    image: mobz/elasticsearch-head:5
    container_name: es-head
    restart: always
    ports:
      - 9100:9100

networks:
   default:
    external:
      name: dakewe

使用X-Pack设置授权加密

按照 Elasticsearch 的要求，如果我们在 docker 的环境中启动 xpack.security.enabled，我们必须也启动 xpack.security.transport.ssl.enabled。否则，我们将会看到如下的错误信息：

[1]:Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]

接下来，针对7.13.0版本的ElasticSearch配置加密授权访问，下边的步骤是必不可少的，建议认真阅读下去。X-Pack 是 ElasticSearch 的一个插件，这个插件将提供与ElasticSearch来往的安全性。通过安装这个插件，我们就可以对 ElasticSearch 的集群节点生成证书，配置服务访问密码，以及使用TLS来确保HTTP客户端与集群之间的通信是加密的。

1
docker exec -it es01 bash

进入容器后，前往工作目录下（即/usr/share/elasticsearch），为Elasticearch集群创建一个证书颁发机构。使用elasticsearch-certutil命令输出一个默认名为elastic-stack-ca.p12的PKCS#12密钥存储库文件，它包含CA的公共证书和用于为每个节点签名证书的私钥。

1
2
cd /usr/share/elasticsearch
bin/elasticsearch-certutil ca

如下的命令来生成一个证书

1
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

上面的命令将使用我们的 CA 来生成一个证书 elastic-certificates.p12:
执行exit退出容器，我们把上面的 elastic-certificates.p12 证书移至./elasticsearch/config/certs文件夹。

1
2
3
docker cp es01:/usr/share/elasticsearch/elastic-certificates.p12 ./elasticsearch/config/certs
sudo chmod -R 777 ./elasticsearch/config
docker-compose down

在docker-compose.yaml配置好证书映射
HUW30H
别忘了docker-compose down关掉服务，因为我们要改配置了。

接下来修改config/elasticsearch.yml来使用加密授权。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
cluster.name: "docker-cluster"
network.host: 0.0.0.0

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type


xpack.license.self_generated.type: basic

xpack.security.enabled: true

# 传输层通信:传输协议用于Elasticsearch节点之间的内部通信
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

# HTTP层通信:客户端到Elasticsearch集群的通信
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.verification_mode: certificate

xpack.monitoring.collection.enabled: false

verification_mode 我们选择certificate，这个模式不会去检查证书的CN,只验证证书是否是信任机构签名的即可.如果我们需要验证,并且配置了IP，则需要把这个模式该为full

如果证书是PEM格式，则使用下方配置

1
2
3
4
5
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.key: /home/es/config/node01.key 
xpack.security.transport.ssl.certificate: /home/es/config/node01.crt 
xpack.security.transport.ssl.certificate_authorities: [ "/home/es/config/ca.crt" ]

设置授权访问的账号和密码

再次启动并进入容器docker exec -it es01 bash，使用elasticsearch-setup-passwords为各个角色创建随机的密码：

1
bin/elasticsearch-setup-passwords auto

也可以使用密码设置来为每个角色设定密码：

1
bin/elasticsearch-setup-passwords interactive

4YqC8I
用interactive参数为每个角色设定密码

访问localhost:9200，输入user elastic的密码，成功获取正确json说明x-pack授权加密已经成功。

为 Elasticsearch 设置认证

内置用户
elastic : 内置的超级用户
kibana_system: 用户Kibana用于连接Elasticsearch并与之通信
logstash_system: Logstash写入监控数据时所需要的ES权限用户
beats_system： Beats写入监控数据时所需要的ES权限用户
apm_system: APM写入监控数据时所需要的ES权限用户
remote_monitoring_user: Metricbeat用户在Elasticsearch中收集和存储监视信息时使用。

使Kibana应用帐号密码

在kibana部分的kibana.yml追加参数：

1
2
elasticsearch.username: "kibana_system"
elasticsearch.password: "XXX"

使logstash应用帐号密码

1
2
3
4
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "https://es0:9200" ]
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "XXX"

然后执行docker-compose up -d kibana 启动服务，等待几分钟，访问localhost:5601，成功出现需要输入密码的界面说明配置成功。
q295bz

到这里，已经完成elasticsearch和kibana的全部部署工作.
pyo0I0

专题目录

ElasticStack-安装篇
ElasticStack-elasticsearch篇
ElasticStack-logstash篇
elasticSearch-mapping相关
elasticSearch-分词器介绍
elasticSearch-分词器实践笔记
elasticSearch-同义词分词器自定义实践
docker-elk集群实践
filebeat与logstash实践
filebeat之pipeline实践
Elasticsearch 7.x 白金级 破解实践
elk的告警调研与实践