настроен filebeat. без подключенного модуля nginx логи приходят в elasticsearch. Так же логи приходили, когда pipeline модуля nginx использовался по умолчанию. Получал ошибки
Provided Grok expressions do not match field value:
. В Grok Debugger выяснил и исправил ошибку парсинга, после внесения изменения правок в pipeline и обновления его в ingest логи толи прекратили поступать либо перестали отображаться в kibana.
filebeat запускал так:
./filebeat -c filebeat.yml --modules=nginx
В логах
2019-03-14T18:19:31.236+0300 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":710,"time":713},"total":{"ticks":2640,"time":2651,"value":2640},"user":{"ticks":1930,"time":1938}},"info":{"ephemeral_id":"3a00ee7c-f02e-4c30-8efc-9ea00ab9fe73","uptime":{"ms":840022}},"memstats":{"gc_next":4249088,"memory_alloc":2199672,"memory_total":86654152,"rss":16384}},"filebeat":{"events":{"added":30,"done":30},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":30,"batches":10,"total":30},"read":{"bytes":3633},"write":{"bytes":22440}},"pipeline":{"clients":5,"events":{"active":0,"published":30,"total":30},"queue":{"acked":30}}},"registrar":{"states":{"current":2,"update":30},"writes":10},"system":{"load":{"1":0.07,"15":0.17,"5":0.11,"norm":{"1":0.0022,"15":0.0053,"5":0.0034}}}}}}
Пробовал запускать так:
./filebeat -c filebeat.yml
В логах filebeat пусто
мои настройки:
filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /somelog/access.log
loging.level: debug
output.elasticsearch:
hosts: ["host:9200"]
setup.kibana:
host: "host:5601"
username: "kibana"
password: "kibanamama"
filebeat.config.modules:
enabled: true
path: ${path.config}/modules.d/*.yml
paths:
- /somelog/access.log
Указан фейковый, т.к. правильные указаны в настройках моудля nginx
../modules.d/nginx.yml
- module: nginx
# Access logs
access:
enabled: true
var.paths:
- /opt/ibm/appuser/nginx/nginx-1.14.2/logs/access.log
# Error logs
error:
enabled: true
var.paths:
- /opt/ibm/appuser/nginx/nginx-1.14.2/logs/error.log
default.json{
"description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"\"?%{IP:nginx.access.remote_ip} - \\[%{TIMESTAMP_ISO8601:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.remote_ip_list}\""
],
"pattern_definitions": {
"IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
},
"ignore_missing": true
}
}, {
"split": {
"field": "nginx.access.remote_ip_list",
"separator": "\"?,?\\s+"
}
}, {
"script": {
"lang": "painless",
"inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { return true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return false; } catch (Exception e) { return false; } } def found = false; for (def item : ctx.nginx.access.remote_ip_list) { if (!isPrivate(item)) { ctx.nginx.access.remote_ip = item; found = true; break; } } if (!found) { ctx.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
}
}, {
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "nginx.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:HH:mm:ss Z"]
}
}, {
"remove": {
"field": "nginx.access.time"
}
}, {
"user_agent": {
"field": "nginx.access.agent",
"target_field": "nginx.access.user_agent"
}
}, {
"remove": {
"field": "nginx.access.agent"
}
}, {
"geoip": {
"field": "nginx.access.remote_ip",
"target_field": "nginx.access.geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
Фармат логов access.log
10.127.238.137 - [2019-03-14T17:54:23+03:00] \"POST /nba/checkNewCalls HTTP/1.1\" 200 30 \"https://somhost:8453/test/index?tab=currentCommuni..." \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36\" \"-\"
