flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <gshap...@cloudera.com>
Subject Re: [KafkaSource] Write timestamp in the header
Date Tue, 14 Apr 2015 18:09:27 GMT
Thanks! I didn't remember the source does that :)

Actually, the best way to insert your own timestamp is using an
interceptor - because the custom code will need to get the information
from within the Kafka message. Your interceptor can overwrite the
default header with the timestamp you prefer.

On Tue, Apr 14, 2015 at 10:15 AM, Tao Li <litao.buptsse@gmail.com> wrote:
> @Gwen Shapira, I think the feature customarize timestamp is useful,
> especialy with HDFSEventSink. Do you consider to add this feature in the
> future?
>
> 2015-04-15 0:55 GMT+08:00 Tao Li <litao.buptsse@gmail.com>:
>>
>> Code like this, KafkaSource will simply put "System.currentTimeMillis()"
>> as "timestamp"
>>
>> 2015-04-15 0:52 GMT+08:00 Gwen Shapira <gshapira@cloudera.com>:
>>>
>>> Are you sure KafkaSource does that? Or do you have an Interceptor?
>>>
>>> On Tue, Apr 14, 2015 at 9:50 AM, Tao Li <litao.buptsse@gmail.com> wrote:
>>> > Hi all:
>>> >
>>> > I found that KafkaSource will put "System.currentTimeMillis()" as
>>> > "timestamp" field in header. If we use HDFSEventSink, the HDFS log can
>>> > be
>>> > rotate by the timestamp.
>>> >
>>> > But the web server log usually contains "timestamp" string, and it
>>> > indicate
>>> > the true timestamp, other than "System.currentTimeMillis()". So I think
>>> > if
>>> > have a feature to allow to customarize the "timestamp" field, will be
>>> > better.
>>
>>
>

Mime
View raw message