在我的场景中,Logstash 接收的系统日志行的“时间戳”采用 UTC 格式,我们在 Elasticsearch 输出中使用事件“时间戳”:
output {
elasticsearch {
embedded => false
host => localhost
port => 9200
protocol => http
cluster => 'elasticsearch'
index => "syslog-%{+YYYY.MM.dd}"
}
}
我的问题是,在 UTC 午夜,Logstash 在一天结束之前将日志发送到不同时区(GMT-4 => 美国/蒙特利尔)的不同索引,并且由于“时间戳”,该索引在 20 小时(晚上 8 点)之后没有日志“ 是 UTC。
我们已经完成了转换时区的工作,但我们遇到了显着的性能下降:
filter {
mutate {
add_field => {
# Create a new field with string value of the UTC event date
"timestamp_zoned" => "%{@timestamp}"
}
}
date {
# Parse UTC string value and convert it to my timezone into a new field
match => [ "timestamp_zoned", "yyyy-MM-dd HH:mm:ss Z" ]
timezone => "America/Montreal"
locale => "en"
remove_field => [ "timestamp_zoned" ]
target => "timestamp_zoned_obj"
}
ruby {
# Output the zoned date to a new field
code => "event['index_day'] = event['timestamp_zoned_obj'].strftime('%Y.%m.%d')"
remove_field => [ "timestamp_zoned_obj" ]
}
}
output {
elasticsearch {
embedded => false
host => localhost
port => 9200
protocol => http
cluster => 'elasticsearch'
# Use of the string value
index => "syslog-%{index_day}"
}
}
有没有办法优化这个配置?
这是优化配置,请尝试测试一下性能。
你不需要使用mutate
and date
插入。使用ruby
直接插件。
input {
stdin {
}
}
filter {
ruby {
code => "
event['index_day'] = event['@timestamp'].localtime.strftime('%Y.%m.%d')
"
}
}
output {
stdout { codec => rubydebug }
}
输出示例:
{
"message" => "test",
"@version" => "1",
"@timestamp" => "2015-03-30T05:27:06.310Z",
"host" => "BEN_LIM",
"index_day" => "2015.03.29"
}
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)