04ELK基本使用-elasticsearch

阅读量: zyh 2019-12-16 14:00:10
Categories: > Tags:

准备

https://www.elastic.co/guide/en/elasticsearch/reference/7.15/system-config.html

# elasticsearch 具体安装命令和版本请以下载页中对应的docker安装方式页里命令为基准
sysctl -a | egrep "(vm.max_map_count|net.ipv4.tcp_retries2)"  # 查看是否过小, 如果过小执行下一条
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_retries2=5' >> /etc/sysctl.conf
sysctl -p

## https://www.elastic.co/guide/en/elasticsearch/reference/7.15/setting-system-settings.html
cat >> /etc/security/limits.conf << EOF
  *           soft   nofile       102400
  *           hard   nofile       102400
  *           soft   nproc        102400
  *           hard   nproc        102400
  *           soft  memlock      unlimited
  *           hard  memlock      unlimited
EOF

## 关闭swap 同时应该在 /etc/fstab 中取消 swap 的挂载
sudo swapoff -a

安装

https://mirrors.huaweicloud.com/elasticsearch/

esVersion=7.15.1
cd /export/src
curl "https://mirrors.huaweicloud.com/elasticsearch/${esVersion}/elasticsearch-${esVersion}-linux-x86_64.tar.gz" -o es.tgz
tar xf es.tgz
mv elasticsearch-${esVersion} ../elasticsearch
useradd -U elasticsearch
chown -R elasticsearch:elasticsearch /export/elasticsearch
su - elasticsearch
cd /export/elasticsearch

配置

https://www.elastic.co/guide/en/elasticsearch/reference/7.15/important-settings.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.15/configuring-stack-security.html

角色

角色介绍的官方文档:https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles

elasticsearch 可以通过node.roles: [xxx]来设定节点角色。如果不设置node.roles,则默认节点拥有所有角色。

角色可分为:master、data和data_xxx系列、ingest、ml、remote_cluster_client、transform。

💢在一个集群中,master和data是必须的角色。

一份通用节点配置

配置中discovery.seed_hosts替换成所有节点的hostname

cp config/elasticsearch.yml config/elasticsearch.yml.default
hostName=`hostname`
cat > /export/elasticsearch/config/elasticsearch.yml << EOF
cluster.name: es-cluster
discovery.seed_hosts: xxx1,xxx2,xxx3
node.name: ${hostName}
#node.master: true
#node.data: false
http.port: 9200
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
resource.reload.interval.high: 5s
EOF

ℹ️xpack.security.transport.ssl.keystore.path、xpack.security.transport.ssl.truststore.path指定的文件名与下面启用TLS中的默认文件名一致。

💁 如果是单节点,则应该加入discovery.type: single-node,并删除discovery.seed_hosts

启用TLS

当启用了xpack.security.enabled: true后,多节点生产模式集群必须启用TLS,否则无法启动集群。

创建CA证书和节点证书

./bin/elasticsearch-certutil ca
=>回车,接受默认文件名 elastic-stack-ca.p12
=>输入CA证书密码
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
=>输入CA证书密码
=>回车,接受默认文件名 elastic-certificates.p12
=>输入节点证书密码(可选)

将生成的elastic-certificates.p12复制到每一个节点的config目录中,并确保授权600

💔我添加了节点证书密码后,ES校验证书失败,不知原因。

💢如果节点证书开启了密码,则需要存在ES中,通过下列命令存储,会存放在config/elasticsearch.keystore

./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
=>输入节点证书密码
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
=>输入节点证书密码

启动、关闭

启动

ES默认会自动调整JVM参数,非必要情况下无需调整.

命令中-Ecluster.initial_master_nodes替换成所有节点的hostname

## export ES_JAVA_OPTS="$ES_JAVA_OPTS xxx"
export ES_TMPDIR="/export/elasticsearch/tmp"
mkdir -p /export/elasticsearch/tmp
su - elasticsearch
cd /export/elasticsearch && ./bin/elasticsearch -d -p es.pid -Ecluster.initial_master_nodes=data01,data02,data03

💢-Ecluster.initial_master_nodes=data01,data02,data03 仅当首次启动的时候需要添加。当集群成功启动后,应删除此配置。重启集群、新节点入集群都不应该加入此配置。

关闭

su - elasticsearch
cd /export/elasticsearch && pkill -F es.pid

内置用户密码生成

通过设置用户密码后,kibana就需要配置elastic账户来访问web。

elastic是超级用户;

logstash_system 是一个监控用户,仅用于 Elastic Stack 监控功能监控 Logstash 实例,并将监控数据存储在安全的 Elasticsearch 集群中,不可用于logstash.output。

✨关于logstash.output去操作ES所需的索引写权限,可以通过 https://www.elastic.co/guide/en/logstash/current/ls-security.html 文档的介绍创建。创建方式可以通过ES API或者kibana控制台。

关于内置用户对应的权限,可以看相关内置角色的权限范围:

https://www.elastic.co/guide/en/elasticsearch/reference/7.15/built-in-roles.html

创建命令如下:

./bin/elasticsearch-setup-passwords auto

Changed password for user apm_system
PASSWORD apm_system = va40HqefLmamecrNSUuv

Changed password for user kibana_system
PASSWORD kibana_system = 34hem76rnIB1XFv6oWGp

Changed password for user kibana
PASSWORD kibana = 34hem76rnIB1XFv6oWGp

Changed password for user logstash_system
PASSWORD logstash_system = KEx2vLvyUu4cY5KOK95s

Changed password for user beats_system
PASSWORD beats_system = 86ZVXoxinMESc5iR9vyA

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = yFTGkenceHBnZFOS6rX5

Changed password for user elastic
PASSWORD elastic = Fb5wzMz7UJTCJJNipJfz

✨设置了elastic用户密码后,你将无法再次运行bin/elasticsearch-setup-passwords命令进行设置。

校验集群状态

使用elastic用户访问健康状态接口。

curl --user elastic:Fb5wzMz7UJTCJJNipJfz -XGET 'http://data01:9200/_cluster/health?pretty'
{
  "cluster_name" : "es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 2,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

在这之后,你可以通过kibana从而方便的进行图形化用户管理.

备份集群

https://www.elastic.co/guide/en/elasticsearch/reference/7.15/backup-cluster.html

问题点

  1. 如果磁盘不够用,则因添加新节点,而不是添加新磁盘路径
  2. cluster.name是集群的唯一标识,任何不一致的cluster.name节点无法加入集群
  3. node.name是节点的唯一标识名,默认取值主机名
  4. network.host的设置是ES判断开发模式和生产模式的依据,默认值是回环地址,ES将认为是开发模式。
  5. 没有data角色的节点如果在启动时在磁盘上找到任何分片数据将拒绝启动,而没有master角色和data角色的节点如果在启动时磁盘上有任何索引元数据将拒绝启动。