epcim
10/26/2017 - 2:00 PM

Fluentd v0.12 converting long epoch (milliseconds) to Date Time string with milleseconds precision

Fluentd v0.12 converting long epoch (milliseconds) to Date Time string with milleseconds precision

# Consider the record contains the time stamp of the event in a record key called 'timestamp'
# e.g. "timestamp": "1502217900063"

# The below will add a new record called `formatted_date` that will include an iso8601(3) formatted date string with milliseconds,
# the trick was to extract from the long epoch value the seconds & remaining milliseconds and convert it to microseconds since Time.at() accepts:
#  `Time.at(seconds, microseconds_with_frac) → time`

<filter tag.*>
  @type record_modifier
  <record>
    formatted_date ${Time.at(record['timestamp'].to_i/1000, record['timestamp'].to_i%1000*1000).utc.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}
  </record>
</filter>

# If you use the fluent-plugin-elasticsearch, you can tell the plugin to use `formatted_date` to generate the @timestamp, like so:
<match **>
  type elasticsearch
  time_key formatted_date
  ...
</match>