Custom logstash log line reference field

Custom_logstash_log_line_reference_field.md

Add line reference value to logstash parsed log field

Issue: Diffrents log with same datetime not ordered inside kibana

Solution:

• Add log line reference to filewatch library to be able to know which line is before the over inside the file
• Add this information to the path field to be able to access this value from logstash config file
• Get the value inside the logstash configuration file using a simple grok filter on the path field file: logstash-1.4.2/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb

Modifications

origin

@sincedb = {}
@sincedb_last_write = 0
[...]
@buffers[path].extract(data).each do |line|
  yield(path, line)
end

to

@sincedb = {}
$oldtime = DateTime.now.strftime('%Q').to_i
$currenttime = 0
$messagenumber = 0
@sincedb_last_write = 0
[...]
@buffers[path].extract(data).each do |line|
  $currenttime = DateTime.now.strftime('%Q').to_i
  if $currenttime == $oldtime
    $messagenumber += 1
  else
    $messagenumber = 0
    $oldtime = $currenttime
  end
  $messagenumberstring = sprintf("%0.9d", $messagenumber)
  yield("#{path}||#{$currenttime}#{$messagenumberstring}", line)
end

Usage

1. Add this grok pattern: PATH_AND_POSITION (%{PATH:path}||%{NUMBER:log_ref_id:int})
2. Use this grok filter inside your logstash configuration

grok { match => { "path", "%{PATH_AND_POSITION}" } overwrite => [ "path" ] }

3. Enjoy your new field called log_ref_id

#####log_ref_id field contains two values:

1. Number of microseconds since 1970-01-01 00:00:00 UTC.
2. Incremented int value if many parsed messages inside the same millisecond (9 characters number, 15 becomes 000000015)