1、elk说明
elk全称:
elasticsearch:
是一个分布式、高扩展、高实时的搜索与数据分析引擎;简称es
logstash:
是开源的服务器端数据处理管道,能够同时从多个来源采集数据,转换数据,然后将数据发送到您最喜欢的“存储库”中;如elasticsearch中
kibana:
是为 Elasticsearch设计的开源分析和可视化平台。你可以使用 Kibana 来搜索,查看存储在 Elasticsearch 索引中的数据并与之交互。你可以很容易实现高级的数据分析和可视化,以图标的形式展现出来。
以上三个组件就是常说的elk~
创新互联建站服务紧随时代发展步伐,进行技术革新和技术进步,经过十余年的发展和积累,已经汇集了一批资深网站策划师、设计师、专业的网站实施团队以及高素质售后服务人员,并且完全形成了一套成熟的业务流程,能够完全依照客户要求对网站进行成都网站制作、成都网站建设、建设、维护、更新和改版,实现客户网站对外宣传展示的首要目的,并为客户企业品牌互联网化提供全面的解决方案。
2、快速部署配置elk
1)部署环境:
Centos7,本文基于7.x部署
172.16.0.213 elasticsearch
172.16.0.217 elasticsearch
172.16.0.219 elasticsearch kibana
kibana只要在其中一台部署即可;
2)配置官方yum源
三台均配置repo源
$ cat /etc/yum.repos.d/elast.repo
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
3)安装
$ cat /etc/hosts
172.16.0.213 ickey-elk-213
172.16.0.217 ickey-elk-217
172.16.0.219 ickey-elk-219
$ yum install elasticsearch -y
4)配置
$ cat /etc/elasticsearch/elasticsearch.yml
cluster.name: elk_test ### 集群名
node.name: ickey-elk-217 ### 节点名需要按节点配置
node.master: true
node.data: true
path.data: /var/log/elasticsearch/data
path.logs: /var/log/elasticsearch/logs
network.host: 172.16.0.217 ### 节点ip
transport.tcp.port: 9300
transport.tcp.compress: true
http.port: 9200
http.max_content_length: 100mb
bootstrap.memory_lock: true
discovery.seed_hosts: ["172.16.0.213","172.16.0.217","172.16.0.219"]
cluster.initial_master_nodes: ["172.16.0.213","172.16.0.217","172.16.0.219"]
gateway.recover_after_nodes: 2
gateway.recover_after_time: 5m
gateway.expected_nodes: 3
修改elasticsearch启动内存分配:
$ /etc/elasticsearch/jvm.options 中
-Xms4g
-Xmx4g
内在一般是系统内存80%左右;分别表示预加载内存和最高使用内存
此时启动elasticsearch
$ systemctl elasticsearch start
5)安装kibana
就在219上安装
$ yum install kinbana -y
配置
$ cat /etc/kibana/kibana.yml|egrep -v "(^$|^#)"
server.port: 5601
server.host: "172.16.0.219"
server.name: "ickey-elk-219"
elasticsearch.hosts: ["http://172.16.0.213:9200","http://172.16.0.217:9200","http://172.16.0.219:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "pass"
elasticsearch.requestTimeout: 40000
logging.dest: /var/log/kibana/kibana.log # 日志输出,默认输出到了/var/log/message
i18n.locale: "zh-CN" # 中文界面
详情配置参考:
https://www.elastic.co/guide/cn/kibana/current/settings.html
2、logstash安装配置及实践
上面已经所存储搜索的es和展示及搜索图片化的kibana安装配置完成,数据获取部分就需要logstash和beat这里主要使用到了logstash和filebeat
lostash收集日志比较重量级,配置也相对复杂点;可定制收集的功能也很多,这里除了安装给也常见配置整理:
1)安装
通过yum源安装,安装源同上
yum install logstash -y
logstash需要jdk支持;因此需要先安装配置java jdk版本1.8及以上即可;
这里安装 jdk-8u211-linux-x64.rpm
$cat /etc/profile.d/java.sh
xport JAVA_HOME=/usr/java/latest
export JAVA_BIN=${JAVA_HOME}/bin
export PATH=${PATH}:${JAVA_HOME}/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
export JRE_HOME=/usr/java/latest
安装完成后需要执行 /usr/share/logstash/bin/system-install
Centos 6的系统通过以下方式管理服务
initctl status|start|stop|restart logstash
CentOS7:
systemctl restart logstash
2)实践配置
收集nginx日志:(nginx服务器上执行)
$ cat /etc/logstash/conf.d/nginx-172.16.0.14.conf
input {
file {
path => ["/var/log/nginx/test.log"]
codec => json
sincedb_path => "/var/log/logstash/null"
discover_interval => 15
stat_interval => 1
start_position => "beginning"
}
}
filter {
date {
locale => "en"
timezone => "Asia/Shanghai"
match => [ "timestamp", "ISO8601" ,"yyyy-MM-dd'T'HH:mm:ssZZ" ]
}
mutate {
convert => [ "upstreamtime", "float" ]
}
mutate {
gsub => ["message", "\x", "\\x"]
}
if [user_agent] {
useragent {
prefix => "remote_"
source => "user_agent"
}
}
if [request] {
ruby {
init => "@kname = ['method1','uri1','verb']"
code => "new_event = LogStash::Event.new(Hash[@kname.zip(event.get('request').split(' '))])
new_event.remove('@timestamp')
new_event.remove('method1')
event.append(new_event)"
remove_field => [ "request" ]
}
}
geoip {
source => "clientRealIp"
target => "geoip"
database => "/tmp/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [
"[geoip][coordinates]", "float",
"upstream_response_time","float",
"responsetime","float",
"body_bytes_sent","integer",
"bytes_sent","integer"]
}
}
output {
elasticsearch {
hosts => ["172.16.0.219:9200"]
index => "logstash-nginx-%{+YYYY.MM.dd}"
workers => 1
template_overwrite => true
}
}
注意需要nginx中的日志格式配置为:
log_format logstash '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"host":"$server_addr",'
'"size":$body_bytes_sent,'
'"domain":"$host",'
'"method":"$request_method",'
'"url":"$uri",'
'"request":"$request",'
'"status":"$status",'
'"referer":"$http_referer",'
'"user_agent":"$http_user_agent",'
'"body_bytes_sent":"$body_bytes_sent",'
'"bytes_sent":"$bytes_sent",'
'"clientRealIp":"$clientRealIp",'
'"forwarded_for":"$http_x_forwarded_for",'
'"responsetime":"$request_time",'
'"upstreamhost":"$upstream_addr",'
'"upstream_response_time":"$upstream_response_time"}';
配置成接收syslog
$ cat /etc/logstash/conf.d/rsyslog-tcp.conf
input {
syslog {
type => "system-syslog"
host => "172.16.0.217"
port => 1514
}
}
filter {
if [type] == "system-syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
if [type] == "system-syslog" {
elasticsearch {
hosts => ["172.16.0.217:9200"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
#workers => 1
template_overwrite => true
}
}
}
客户端需要配置:
$ tail -fn 1 /etc/rsyslog.conf
. @172.16.0.217:1514
配置收集硬件日志服务器
[yunwei@ickey-elk-217 ~]$ cat /etc/logstash/conf.d/hardware.conf
input {
syslog {
type => "hardware-syslog"
host => "172.16.0.217"
port => 514
}
}
filter {
if [type] == "hardware-syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
if [type] == "hardware-syslog" {
elasticsearch {
hosts => ["172.16.0.217:9200"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}
}
}
3、filebeat安装配置及应用实践
1)说明
filebeat 原先是基于 logstash-forwarder 的源码改造出来的。换句话说:filebeat 就是新版的 logstash-forwarder,也会是 Elastic Stack 在 shipper 端的第一选择。
下图摘自官方,es logstasilebeat.pngh filbeat kafa redis之间的关系;如图:
2)安装
同样基于以上的yum源
$ yum install filebeat -y
3)配置之收集runtime和php-fpm错误日志
[root@ickey-app-api-52 yunwei]# cat /etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
paths:
- /home/wwwroot/.ickey.cn/runtime/logs/.log
fields:
type: "runtime"
json.message_key: log
json.keys_under_root: true
- type: log
enabled: true
paths:
- /var/log/php-fpm/www-error.log
fields:
type: "php-fpm"
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 2
#============================== Kibana =====================================
setup.kibana:
host: "172.16.0.219:5601"
#============================= Elastic Cloud ==================================
output.elasticsearch:
hosts: ["172.16.0.213:9200","172.16.0.217:9200","172.16.0.219:9200"]
indices:
- index: "php-fpm-log-%{+yyyy.MM.dd}"
when.equals:
fields.type: "php-fpm"
- index: "runtime-log-%{+yyyy.MM.dd}"
when.equals:
fields.type: "runtime"
pipelines:
- pipeline: "php-error-pipeline"
when.equals:
fields.type: "php-fpm"
#================================ Processors =====================================
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
#================================ Logging =====================================
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
说明:
php-fpm error.log格式如下:
[29-Oct-2019 11:33:01 PRC] PHP Fatal error: Call to a member function getBSECollection() on null in /var/html/wwwroot/framework/Excel5.php on line 917
由于 我们需要提取其中的时间,PHP Fatal error 及出错的行数;在logstash中收集需要定义 grok,filebeat则需要通过ingest处理,大概过程 是这样的filebeat先获取内容 放到logstash上 通过ingest定义输出成我们相要的样子;
因此需要在logstash上做如下操作:
[root@ickey-elk-213 ~]# cat phperror-pipeline.json
{
"description": "php error log pipeline",
"processors": [
{
"grok": {
"field": "message",
"patterns": "%{DATA:datatime} PHP .*: %{DATA:errorinfo} in %{DATA:error-url} on line %{NUMBER:error-line}"
}
}
]
}
应用 :
curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_ingest/pipeline/php-error-pipeline' -d@phperror-pipeline.json
查询:
curl -H 'Content-Type: application/json' -GET 'http://localhost:9200/_ingest/pipeline/php-error-pipeline'
删除:
curl -H 'Content-Type: application/json' -XDELETE 'http://localhost:9200/_ingest/pipeline/php-error-pipeline'
收集数据库日志:
filebeat.inputs:
- type: log
paths:
- /var/log/MySQL/mysql.err
fields:
type: "mysqlerr"
exclude_files: ['Note']
multiline.pattern: '^[0-9]{4}.*'
multiline.negate: true
multiline.match: after
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: True
setup.template.settings:
index.number_of_shards: 2
setup.kibana:
host: "172.16.0.219:5601"
output.elasticsearch:
hosts: ["172.16.0.213:9200"]
indices:
- index: "mysql-err-%{+yyyy.MM.dd}"
when.equals:
fields.type: "mysqlerr"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
4、安装配置elasticsearch-head
elasticsearch-head是开源的,图形化查看操作es中索引web界面;
1)安装
$ git clone https://github.com/mobz/elasticsearch-head.git
$ cd elasticsearch-head
$ registry=https://registry.npm.taobao.org
$ npm install grunt -save --
└─┬ grunt@1.0.1
.....省略....
├── path-is-absolute@1.0.1
└── rimraf@2.2.8
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression
$ npm install --registry=https://registry.npm.taobao.org
npm WARN deprecated http2@3.3.7: Use the built-in module in node 9.0.0 or newer, instead
[ ............] - fetchMetadata: verb afterAdd /root/.npm/debug/2.6.9/package/package.json written
此步需要等待一段时间
2)配置开机自启服务
$ cat /usr/bin/elasticsearch-head
#!/bin/bash
# chkconfig: - 25 75
# description: starts and stops the elasticsearch-head
data="cd /usr/local/src/elasticsearch-head/; nohup npm run start > /dev/null 2>&1 & "
START(){
eval $data && echo -e "elasticsearch-head start\033[32m ok\033[0m"
}
STOP(){
ps -ef |grep grunt |grep -v "grep" |awk '{print $2}' |xargs kill -s 9 > /dev/null && echo -e "elasticsearch-head stop\033[32m ok\033[0m"
}
STATUS(){
PID=$(ps aux |grep grunt|grep -v grep|awk '{print $2}')
}
case "$1" in
start)
START
;;
stop)
STOP
;;
restart)
STOP
sleep 3
START
;;
*)
echo "Usage: elasticsearch-head (start|stop|restart)"
;;
esac
访问:
http://172.16.0.219:9100 如图:
当前标题:elk(elasticsearch、logstast,kibana)filebeat部署与实践
文章出自:https://www.cdcxhl.com/article40/jpsiho.html
成都网站建设公司_创新互联,为您提供关键词优化、手机网站建设、建站公司、面包屑导航、定制网站、服务器托管
声明:本网站发布的内容(图片、视频和文字)以用户投稿、用户转载内容为主,如果涉及侵权请尽快告知,我们将会在第一时间删除。文章观点不代表本网站立场,如需处理请联系客服。电话:028-86922220;邮箱:631063699@qq.com。内容未经允许不得转载,或转载时需注明来源: 创新互联