背景
在Logstash读取nginx日志的消息简要如下,可以看到这里的@timestamp和time中的时间戳不一致,那么我们可以修改@timestamp指定为time。
{
"@timestamp" => 2022-07-21T07:12:40.806Z,
"url" => "/index.php?m=message&f=ajaxGetMessage&t=html&windowBlur=1",
"referrer" => "http://192.168.1.240:10003/index.php?m=bug&f=browse&productID=2",
"remote_ip" => "192.168.1.174",
"user_name" => "-",
"method" => "GET",
"response_code" => "200",
"host" => {
"name" => "84032b3b70a8"
},
"time" => "21/Jul/2022:15:12:34 +0800",
"user_agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"http_version" => "1.1",
"body_sent" => "5",
}
配置
配置如下,只需要编辑logstash/conf.d/nginx.conf
文件,修改filter的部分即可:
input {
beats {
port => 5044
codec => "json"
}
}
filter {
grok {
match => { "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:user_agent}\"' }
remove_field => "message"
}
date {
match => ["time", "dd/MMM/yyyy:HH:mm:ss Z"] #匹配timestamp字段
target => "@timestamp" #将匹配到的数据写到@timestamp字段中
}
}
output {
elasticsearch {
hosts => ["192.168.1.240:9200"]
index => "nginx-access-log-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
最后重启Logstash生效即可。