Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 427C917DB8 for ; Tue, 8 Sep 2015 15:41:33 +0000 (UTC) Received: (qmail 36891 invoked by uid 500); 8 Sep 2015 15:41:32 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 36835 invoked by uid 500); 8 Sep 2015 15:41:32 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 36825 invoked by uid 99); 8 Sep 2015 15:41:32 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Sep 2015 15:41:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6D88AC01AB for ; Tue, 8 Sep 2015 15:41:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id KLs1gjYqP7sl for ; Tue, 8 Sep 2015 15:41:31 +0000 (UTC) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id C09F6204D9 for ; Tue, 8 Sep 2015 15:41:30 +0000 (UTC) Received: by lbbmp1 with SMTP id mp1so55865907lbb.1 for ; Tue, 08 Sep 2015 08:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=kYp6+q2i+3Al3RSjxujCu2GLZjgQSMrhXQ5QRizBQKw=; b=LW7rTNkTeVkD8NLD21QnE3VLEZnBFlcJMpsINZlTmXJxryiXgBYfEnA/WUgaN0nprE TpsZwN3oWdAHQVXzeTgIlAph8L6CG6rSV8w4m9svVnh7XJiRgm6Ew2VG71CTP0D3PAQH 8+xi+0j3yQd3IHUUj+xttxahUDMGZjPIt5myZ4jRkTp3mQRjqnzxHL5l6YJYGoH9uGyB 8MSc33fdkybdOdld+RP276b8hFsLt2RDXbIcYQ2MUNx4ESpdjhKaEKxPzCezenMb4FC8 Vbo0Ab1Lp2MEvD11VxOTtaDSJIkWAAxBCammy5B7/BK/mj9zeV6z/1OCkk0baGI9a+BQ 4kug== X-Received: by 10.152.36.161 with SMTP id r1mr23451875laj.82.1441726890165; Tue, 08 Sep 2015 08:41:30 -0700 (PDT) MIME-Version: 1.0 From: IT CTO Date: Tue, 08 Sep 2015 15:41:20 +0000 Message-ID: Subject: Newbe question about Flume-ng - ElasticSearchSink To: user@flume.apache.org Content-Type: multipart/alternative; boundary=089e0158b5e426cbca051f3e338e --089e0158b5e426cbca051f3e338e Content-Type: text/plain; charset=UTF-8 Hi, I just started learning flume and using it to send events to our HDFS cluster and log them in elasticsearch. I am using the latest build Flume 1.6.0 Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080 Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015 >From source with checksum b29e416802ce9ece3269d34233baf43f I found two issues which I think should be filed as bug: 1) When using agent.sinks.elastic-sink.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer the event timestamp is stored as string and not dateOptionalTime if the agent.sinks.elastic-sink.serializer is not set the code works as expected. 2) Documentation BUG - when using elasticsearchsink a timestamp field need to be created mainly if we want to use it with kibana (as the documentation say) so in order for it to work we should use the timestamp interceptor. I think the documentation should note it in the elasticearchsink section. Thanks, Eran -- Eran | "You don't need eyes to see, you need vision" (Faithless) --089e0158b5e426cbca051f3e338e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
I just started learning flume and using it to send= events to our HDFS cluster and log them in elasticsearch.
I am u= sing the latest build=C2=A0

Flume 1.6.0
= =
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled = by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with = checksum b29e416802ce9ece3269d34233baf43f

I = found two issues which I think should be filed as bug:
1) When us= ing=C2=A0
agent.sinks.elastic-sink.serializer =3D org.apache.flum= e.sink.elasticsearch.ElasticSearchDynamicSerializer
the event= timestamp is stored as string and not dateOptionalTime if the agent.sinks.= elastic-sink.serializer is not set the code works as expected.
2) Documentation BUG - when using elasticsearchsink a timestam= p field need to be created mainly if we want to use it with kibana (as the = documentation say) so in order for it to work we should use the timestamp i= nterceptor. I think the documentation should note it in the elasticearchsin= k section.

Thanks,
Eran


--
Era= n | "You don't need eyes to see, you need vision" (Faithless)=
--089e0158b5e426cbca051f3e338e--