Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 97E2F17951 for ; Tue, 14 Apr 2015 18:11:35 +0000 (UTC) Received: (qmail 84398 invoked by uid 500); 14 Apr 2015 18:11:22 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 84345 invoked by uid 500); 14 Apr 2015 18:11:22 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 84335 invoked by uid 99); 14 Apr 2015 18:11:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Apr 2015 18:11:22 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gshapira@cloudera.com designates 209.85.212.173 as permitted sender) Received: from [209.85.212.173] (HELO mail-wi0-f173.google.com) (209.85.212.173) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Apr 2015 18:11:18 +0000 Received: by widjs5 with SMTP id js5so94389588wid.1 for ; Tue, 14 Apr 2015 11:09:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=w6NzCCm9IIigkFNs5Y+vDT5F+ItFt6LYI1OktpxDv/o=; b=QUafVmEKSREvB7lOne8PxcueuXnxvqvy+863Hh9Mza5KxZb/ObcU1hZsSlTI4+TypO TwYdPshFJfeRWoHcM4ITmAbLdIgGYYHTO0ZzyIa1kANEj64ThQw6vRJ8CyWhvRnlhuAz 9niu4gIswped/+1xmJXELYYvpG+qu1A3HE+fQjHuRVrer/i/4UrMDN14T7o16LjPIJ4K awC7Zlpir1KfUcapaGn+ZxVTOO5loA8Mjwvr8YUSlXTormoTK/mbJgor50eoxN+VhD4D CqyiL9iyG/rQx1PXSpiUxVrc1UzeGW0EsVas9Us6bWyzGc1YwijBSsyACUVbLaLd2JTy rV0g== X-Gm-Message-State: ALoCoQlH3TiubMTMGnYQGOVTEVZXMgGzq9Dl+RSWmcn52KHiHK1SFEDNJWQUslDVKEm/mxN6c3fD MIME-Version: 1.0 X-Received: by 10.180.97.164 with SMTP id eb4mr35309030wib.3.1429034967506; Tue, 14 Apr 2015 11:09:27 -0700 (PDT) Received: by 10.27.139.212 with HTTP; Tue, 14 Apr 2015 11:09:27 -0700 (PDT) In-Reply-To: References: Date: Tue, 14 Apr 2015 11:09:27 -0700 Message-ID: Subject: Re: [KafkaSource] Write timestamp in the header From: Gwen Shapira To: user@flume.apache.org Content-Type: text/plain; charset=UTF-8 X-Virus-Checked: Checked by ClamAV on apache.org Thanks! I didn't remember the source does that :) Actually, the best way to insert your own timestamp is using an interceptor - because the custom code will need to get the information from within the Kafka message. Your interceptor can overwrite the default header with the timestamp you prefer. On Tue, Apr 14, 2015 at 10:15 AM, Tao Li wrote: > @Gwen Shapira, I think the feature customarize timestamp is useful, > especialy with HDFSEventSink. Do you consider to add this feature in the > future? > > 2015-04-15 0:55 GMT+08:00 Tao Li : >> >> Code like this, KafkaSource will simply put "System.currentTimeMillis()" >> as "timestamp" >> >> 2015-04-15 0:52 GMT+08:00 Gwen Shapira : >>> >>> Are you sure KafkaSource does that? Or do you have an Interceptor? >>> >>> On Tue, Apr 14, 2015 at 9:50 AM, Tao Li wrote: >>> > Hi all: >>> > >>> > I found that KafkaSource will put "System.currentTimeMillis()" as >>> > "timestamp" field in header. If we use HDFSEventSink, the HDFS log can >>> > be >>> > rotate by the timestamp. >>> > >>> > But the web server log usually contains "timestamp" string, and it >>> > indicate >>> > the true timestamp, other than "System.currentTimeMillis()". So I think >>> > if >>> > have a feature to allow to customarize the "timestamp" field, will be >>> > better. >> >> >