ELK收集Nginx日志教程

重点是通过Logstash正则提取Nginx日志

一、Nginx日志格式配置

log\_format main '$remote\_addr - $remote\_user [$time\_local\] "$request" '

'$status $body\_bytes\_sent "$http\_referer" '

'"$http\_user\_agent" "$http\_x\_forwarded\_for"';

二、Nginx日志格式

192.168.20.7 - - [18/Nov/2020:21:42:26 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; Win64;

x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" "-"

192.168.20.7 - - [18/Nov/2020:21:42:26 +0800] "GET /favicon.ico HTTP/1.1" 404 555 "http://192.168.20.41/" "Mozil

la/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" "

-"

三、内置正则提取语法：%{内置正则表达式:字段名}

%{IP:remote\_addr} - (%{WORD:remote\_user}|-) \[%{HTTPDATE:time\_local}\] "%{WORD:method} %{NOTSPACE:request} HTTP/%{NUMBER}" %{NUMBER:status} %{NUMBER:body\_bytes\_sent} %{QS} %{QS:http\_user\_agent}

四、Logstash正则提取Nginx写入ES

1、使用kibana自身的Grok提取，需要掌握正则表达式

2、使用正式提取语法

3、把正行的nginx日志分段提取出来

4、配置logstash

5、默认logstash收集nginx日志

[root@master nginx]# more /etc/logstash/conf.d/logstash.conf

input {

file {

path => "/var/log/nginx/access.log"

}

}

output {

elasticsearch {

hosts => ["http://192.168.20.41:9200", "http://192.168.20.42:9200"]

user => "elastic"

password => "hahashen"

index => "sjgnginx-%{+YYYY.MM.dd}"

}

}

6、kibana创建Create index pattern

下一步

7、kibana显示nginx日志

8、最后正则拆分nginx日志

请大家扫描关注，共同学习