使用supervisor管理ELK
配置supervisor,使用supervisor启动各个进程,方便管理。同时ELK各个进程使用supervisor管理还有一个较大的优势就是可以很轻松开启多个进程。具体supervisor介绍使用可以看:使用Supervisor管理进程
1)首先各程序都是用elk用户来运行
1 2 3 |
[root@localhost ~]# chown elk.elk /usr/local/kibana-4.5.0-linux-x64/ -R [root@localhost ~]# chown elk.elk /usr/local/logstash-2.3.2/ -R [root@localhost ~]# chown elk.elk /usr/local/elasticsearch-2.3.2/ -R |
2)安装配置supervisor
1 2 |
[root@localhost ~]# yum install python-pip [root@localhost ~]# pip install supervisor |
1 2 |
[root@localhost ~]# mkdir -m 755 -p /etc/supervisor/conf.d/ [root@localhost ~]# echo_supervisord_conf > /etc/supervisor/supervisord.conf |
修改配置文件/etc/supervisor/supervisord.conf添加如下行:
1 2 |
[include] files = /etc/supervisor/conf.d/*.conf |
3)创建各程序的配置文件
第一个:logstash-shipper.conf
1 2 3 4 5 6 7 8 9 10 11 |
[root@localhost ~]# cat /etc/supervisor/conf.d/logstash-shipper.conf [program:logstash-shipper] command = /usr/local/logstash-2.3.2/bin/logstash -f /usr/local/logstash-2.3.2/etc/logstash_shipper.conf autostart = true autorestart = true startsecs = 5 startretries = 3 user = elk redirect_stderr = true stdout_logfile=/data/log/elk/logstash-shipper-std.log stderr_logfile=/data/log/elk/logstash-shipper-err.log |
第二个:logstash-indexer.conf
1 2 3 4 5 6 7 8 9 10 11 |
[root@localhost ~]# cat /etc/supervisor/conf.d/logstash-indexer.conf [program:logstash-indexer] command = /usr/local/logstash-2.3.2/bin/logstash -f /usr/local/logstash-2.3.2/etc/logstash_indexer.conf autostart = true autorestart = true startsecs = 5 startretries = 3 user = elk redirect_stderr = true stdout_logfile=/data/log/elk/logstash-indexer-std.log stderr_logfile=/data/log/elk/logstash-indexer-err.log |
第三个:elastsearch-node1.conf
1 2 3 4 5 6 7 8 9 10 11 |
[root@localhost ~]# cat /etc/supervisor/conf.d/elastsearch-node1.conf [program:elasticsearch-node1] command = /usr/local/elasticsearch-2.3.2/bin/elasticsearch autostart = true autorestart = true startsecs = 5 startretries = 3 user = elk redirect_stderr = true stdout_logfile=/data/log/elk/elasticsearch-node1-std.log stderr_logfile=/data/log/elk/elasticsearch-node1-err.log |
第四个:kibana.conf
1 2 3 4 5 6 7 8 9 10 11 |
[root@localhost ~]# cat /etc/supervisor/conf.d/kibana.conf [program:kibana] command = /usr/local/kibana-4.5.0-linux-x64/bin/kibana autostart = true autorestart = true startsecs = 5 startretries = 3 user = elk redirect_stderr = true stdout_logfile=/data/log/elk/kibana-std.log stderr_logfile=/data/log/elk/kibana-err.log |
具体配置文件含义还是看:使用Supervisor管理进程
创建日志存放路径
1 |
[root@localhost ~]# mkdir -p /data/log/elk/ |
4)启动supervisord
1 |
[root@localhost ~]# supervisord -c /etc/supervisor/supervisord.conf |
5)查看启动结果
1 2 3 4 5 |
[root@localhost ~]# supervisorctl status elasticsearch-node1 RUNNING pid 7828, uptime 0:14:33 kibana RUNNING pid 7780, uptime 0:14:37 logstash-indexer RUNNING pid 7790, uptime 0:14:36 logstash-shipper RUNNING pid 7826, uptime 0:14:34 |
各个进程状态都是RUNNING,进程都运行正常了。如果有报错去查看日志即可,常用管理命令有如下这些:
1 2 3 4 5 |
supervisorctl status supervisorctl shutdown supervisorctl update supervisorctl restart [NAME] supervisorctl stop [NAME] |
使用supervisor开启多个进程
一般当日志量太大的时候,少量的indexer可能无法很快地消费redis队列。所以,当发现数据有堆积的时候,就立即可以利用supervisor启动多个indexer进程。主要添加如下两行:
1 2 |
process_name=%(process_num)s numprocs=3 |
这里表示开启为3个进程数,整体配置如下:
1 2 3 4 5 6 7 8 9 10 11 12 |
[program:logstash-indexer] process_name=%(process_num)s numprocs=3 command = /usr/local/logstash-2.3.2/bin/logstash -f /usr/local/logstash-2.3.2/etc/logstash_indexer.conf autostart = true autorestart = true startsecs = 5 startretries = 3 user = elk redirect_stderr = true stdout_logfile=/data/log/elk/logstash-indexer-std.log stderr_logfile=/data/log/elk/logstash-indexer-err.log |
更新一下supervisor,看看结果。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
[root@localhost ~]# supervisorctl elasticsearch-node1 RUNNING pid 8747, uptime 2:05:04 kibana RUNNING pid 8007, uptime 3:02:59 logstash-indexer RUNNING pid 8008, uptime 3:02:59 logstash-shipper RUNNING pid 8005, uptime 3:02:59 supervisor> update logstash-indexer: stopped logstash-indexer: updated process group supervisor> status elasticsearch-node1 RUNNING pid 8747, uptime 2:05:21 kibana RUNNING pid 8007, uptime 3:03:16 logstash-indexer:0 STARTING logstash-indexer:1 STARTING logstash-indexer:2 STARTING logstash-shipper RUNNING pid 8005, uptime 3:03:16 supervisor> status elasticsearch-node1 RUNNING pid 8747, uptime 2:05:24 kibana RUNNING pid 8007, uptime 3:03:19 logstash-indexer:0 RUNNING pid 8970, uptime 0:00:07 logstash-indexer:1 RUNNING pid 8969, uptime 0:00:07 logstash-indexer:2 RUNNING pid 8971, uptime 0:00:07 logstash-shipper RUNNING pid 8005, uptime 3:03:19 |
可以看到,indexer的进程里面开启为3个。效率是不是很快。