Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4BC701115B for ; Thu, 10 Apr 2014 15:13:53 +0000 (UTC) Received: (qmail 25023 invoked by uid 500); 10 Apr 2014 15:13:52 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 24446 invoked by uid 500); 10 Apr 2014 15:13:50 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 24430 invoked by uid 99); 10 Apr 2014 15:13:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 15:13:48 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of simeon_zaharici@yahoo.com designates 98.138.120.51 as permitted sender) Received: from [98.138.120.51] (HELO nm44.bullet.mail.ne1.yahoo.com) (98.138.120.51) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 10 Apr 2014 15:13:41 +0000 Received: from [127.0.0.1] by nm44.bullet.mail.ne1.yahoo.com with NNFMP; 10 Apr 2014 15:13:18 -0000 Received: from [98.138.100.102] by nm44.bullet.mail.ne1.yahoo.com with NNFMP; 10 Apr 2014 15:13:07 -0000 Received: from [66.196.81.173] by tm101.bullet.mail.ne1.yahoo.com with NNFMP; 10 Apr 2014 15:13:07 -0000 Received: from [98.139.212.238] by tm19.bullet.mail.bf1.yahoo.com with NNFMP; 10 Apr 2014 15:13:07 -0000 Received: from [127.0.0.1] by omp1047.mail.bf1.yahoo.com with NNFMP; 10 Apr 2014 15:13:07 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 29872.56813.bm@omp1047.mail.bf1.yahoo.com Received: (qmail 63378 invoked by uid 60001); 10 Apr 2014 15:13:06 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1397142786; bh=PJ/O8NVazmTPNrffld+WV+OqG45Qib9afWvs8Jb2skU=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=OHWziN8beUuU7mb2Ic2Ck8HBeHYpA/VLm3bP6spXthEGuoU93HAlav9WtA9fiv6x1PFweG8j4C9N5c+ujyI9q2+PX4ksK2NmA7Ut6CJIaZc4GjAVELRe3+bOu47kk9e3beHyryo/MHxPovrBe/opjSHX1KfnMSyeuscUfcxh6YU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=LzIdIo4oxmgQLwt12iNPqAEHOlr0o64noGDMwwZ5Z3+KJN2HtrRdIqW1m/3bO/fZgNX0woTcU5tuja/gp04a2aAPWONIuaiGErJW+ztnslUXIZVTRiwvUIGey6mWFMu24vKXzRU4wQa9Rqnh5X54Su5ZCpYe7Kzj31ovzkG4uX0=; X-YMail-OSG: F8Q7arEVM1mlBCwKyKFHKDE07Cxc_DQhn0xYS.IfgISLh4A iLSRRtZA2snijqcoac5vR16dIodKb7othGXwaK5sgjb5q3p1APiFcmIcYhlo 0sz_G9HhevFY.GO5C1vq..yvKXU326ZY3RoKyllSGwJadNfMAuds1sP.4Bqa lUv61mPX5iVmGhPt3C7plKSm3ewia5nKeu2Uihs6AcckCX_chSZ5bAtZ_lJo l4MyU7FKmkPS7l1PGZmqjkugPoEUH_Y6PzvMCHsOiWpKA8EIGV82sKgm_eFp 0mSgdz.NPGky0QLuP7OwvO81yaowplaCUnVCrStneLAWJO4WzhrBtl29z3Vr WxvxJe19Sb1FTwz0mLO48OU7ClCEGeNjTUd1lfzga60OAOm2hGbX2R1DKf58 NGtk17vOXzYzhbwjDJBj5LjJYBOVUpYwvK59aIZT_du_JVfdA9uBiZNVGzbc i7qALjx6EW6h0E06VpXWUa3w9JfMIGlMSSycHSccxtGWRsUcCahnBxon7NLC HBHIYSIfv9DYdeh56H35Vq_wnEros_zUZQVM4uTJFOrk22e4BnV.78NVb60m jZxTZTg-- Received: from [184.144.197.47] by web140805.mail.bf1.yahoo.com via HTTP; Thu, 10 Apr 2014 08:13:06 PDT X-Rocket-MIMEInfo: 002.001,SGVsbG8KCkkgZm9sbG93ZWQgdGhlIGluc3RydWN0aW9ucyBoZXJlwqBodHRwczovL2ZsdW1lLmFwYWNoZS5vcmcvRmx1bWVVc2VyR3VpZGUuaHRtbCNlbGFzdGljc2VhcmNoc2luayBhbmQgSSBhbSB1c2luZyBhIDAuOTAuNyBqYXIgb24gdGhlIGZsdW1lIHNlcnZlciBhZ2FpbnN0IGFuIGVsYXN0aWMgc2VhcmNoIGNsdXN0ZXIgcnVubmluZyAwLjkwLjcuCgpBbHNvIHNlbmRpbmcgcGxhaW4gdGV4dCB0aHJvdWdoIHRoZSBuZXRjYXQgc291cmNlIHdvcmtzIGZpbmUuIFRoZSBvbmx5IHByb2JsZW0gaXMgd2hlbiABMAEBAQE- X-Mailer: YahooMailWebService/0.8.182.648 References: <1394041976.89976.YahooMailNeo@web140802.mail.bf1.yahoo.com> <1397091497.40185.YahooMailNeo@web140803.mail.bf1.yahoo.com> <1397092679.57238.YahooMailNeo@web140803.mail.bf1.yahoo.com> <49FFBB76-D820-4F8F-BAA4-5DD8420C2B20@apache.org> Message-ID: <1397142786.99338.YahooMailNeo@web140805.mail.bf1.yahoo.com> Date: Thu, 10 Apr 2014 08:13:06 -0700 (PDT) From: Simeon Zaharici Reply-To: Simeon Zaharici Subject: Re: Json over netcat source To: "user@flume.apache.org" In-Reply-To: <49FFBB76-D820-4F8F-BAA4-5DD8420C2B20@apache.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="1230856681-210918173-1397142786=:99338" X-Virus-Checked: Checked by ClamAV on apache.org --1230856681-210918173-1397142786=:99338 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello=0A=0AI followed the instructions here=C2=A0https://flume.apache.org/F= lumeUserGuide.html#elasticsearchsink and I am using a 0.90.7 jar on the flu= me server against an elastic search cluster running 0.90.7.=0A=0AAlso sendi= ng plain text through the netcat source works fine. The only problem is whe= n the text that is sent is formatted as json=0A=0AIf I send the words plain= text over the netcat source it ends up like this in elasticsearch=0A{"@mes= sage":"plain text\r","@timestamp":"2014-04-10T14:16:38.113Z","@fields":{"ti= mestamp":"1397139398113"}}=0A=0AIf I send { =E2=80=9Cplain=E2=80=9D : =E2= =80=9Ctext=E2=80=9D }=0A=0A{"@message":"org.elasticsearch.common.xcontent.X= ContentBuilder@4710ad01","@timestamp":"2014-04-10T14:20:00.871Z","@fields":= {"timestamp":"1397139600871"}}=0A=0AAlso sending just {} over the netcat so= urce results in the same error.=0A=0AThanks,=0ASimeon=0A=0AOn Thursday, Apr= il 10, 2014 2:48:49 AM, Mike Percy wrote:=0A =0ANot sur= e either but make sure you're using a compatible version of ElasticSearch.= =0A=0ASent from my iPhone=0A=0AOn Apr 9, 2014, at 9:43 PM, Hari Shreedharan= wrote:=0A=0A=0AThen I really don't know what t= he issue is. Someone more familiar with elastic search sink will need to lo= ok at it.=0A>=0A>=0A>Hari=0A>=0A>On Wednesday, April 9, 2014, Simeon Zahari= ci wrote:=0A>=0A>Hello=0A>>=0A>>=0A>>This is to= be able to easily integrate log centralization clients such as nxlog with = flume=C2=A0=0A>>=0A>>=0A>>The problem is not related to the splitting, or t= he size, even a message like { "blah": "blah" } will result in the exceptio= n "org.elasticsearch.common.xcontent.XContentBuilder"=0A>>=0A>>=0A>>Simeon= =0A>>On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan wrote:=0A>> =0A>>Why exactly do you want to send JSON over n= etcat? Why don't you simply use the Avro Source with Flume SDK? I am not en= tirely sure about the Elastic search sink. The reason your JSON is not work= ing is likely because netcat is splitting the events per newline or when it= reaches a fixed size.=C2=A0=0A>>=0A>>=0A>>=0A>>=0A>>Hari=0A>>=0A>>=0A>>=0A= >>On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici wrote:=0A>>=0A>>Hello=0A>>>=0A>>>=0A>>>No, I did not find any solution t= o this=0A>>>=0A>>>=0A>>>Simeon=0A>>>On Monday, April 7, 2014 6:09:21 AM, De= epak Subhramanian wrote:=0A>>> =0A>>>Hi Sim= eon,=0A>>>I am facing the same problem. Did you find any solution ?=0A>>>= =0A>>>=0A>>>Thanks, Deepak=C2=A0=0A>>>=0A>>>=0A>>>=0A>>>On Wed, Mar 5, 2014= at 5:52 PM, Simeon Zaharici wrote:=0A>>>=0A>>>= Hi guys=0A>>>>=0A>>>>=0A>>>>I would like to send json data to a flume netca= t source and then send it to an elasticsearch sink that uses the =0A>>>>org= .apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.=0A>>= >>=0A>>>>It works to send plain text on the netcat source, however when I s= end json the event ends up in elasticsearch as 'org.elasticsearch.common.xc= ontent.XContentBuilder'.=0A>>>>=0A>>>>Which would be the easiest way to go = about this ? I saw from reading docs that creating a morphline interceptor = could be the way to go but I did not fully understand how that=0A works.=0A= >>>>=0A>>>>Thanks a lot=0A>>>>=0A>>>=0A>>>=0A>>>=0A>>>-- =0A>>>Deepak Subhr= amanian =0A>>>=0A>>>=0A>>=0A>>=0A>> --1230856681-210918173-1397142786=:99338 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hello

I followed the instructions here https://flume.apache.org/Flum= eUserGuide.html#elasticsearchsink and I am using a 0.90.7 jar on the flume = server against an elastic search cluster running 0.90.7.

Also sending plain text through the netc= at source works fine. The only problem is when the text that is sent is formatted as json

If = I send the words plain text over the netcat source it ends up like this in = elasticsearch
{"@message":"plain text\r","@= timestamp":"2014-04-10T14:16:38.113Z","@fields":{"timestamp":"1397139398113= "}}

If I send { =E2=80= =9Cplain=E2=80=9D : =E2=80=9Ctext=E2=80=9D }

{"@message":"org.elasticsearch.common.xcontent.XContentBuild= er@4710ad01","@timestamp":"2014-04-10T14:20:00.871Z","@fields":{"timestamp"= :"1397139600871"}}

Also se= nding just {} over the netcat source results in the same error.

Thanks,
=0A=0A=0A=0A=0A=0A= =0A=0A=0A=0A=0A=0A=0A=0A=0A=0A
Simeon

On Thursday, April 10= , 2014 2:48:49 AM, Mike Percy <mpercy@apache.org> wrote:
 =
= Not sure either but make sure you're using a compatible version of ElasticS= earch.

Sent from my iPhone

On Apr 9, 2014, at 9:43 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:

Then= I really don't know what the issue is. Someone more familiar with elastic = search sink will need to look at it.

Hari=

On Wednesday, April 9, 2= 014, Simeon Zaharici <simeon_zaharici@yahoo.com> wrote:
= =0A
=0AHello

=0A
This is to be able to easily integrate log centralization clients = such as nxlog with flume 
=0A

=0AThe p= roblem is not related to the splitting, or the size, even a message like { = "blah": "blah" } will result in the exception "org.elasticsearch.common.xcontent.XContentB= uilder"
=0A

=0A
Simeon
=0A
=0A <= div dir=3D"ltr"> On Wednesday, April 9, 2014 9:09:18 = PM, Hari Shreedharan <hshre= edharan@cloudera.com> wrote:
=0A
<= div>
Why exactly do you want to send JSON over ne= tcat? Why don't you simply use the Avro Source with Flume SDK? I am not ent= irely sure about the Elastic search sink. The reason your JSON is not worki= ng is likely because netcat is splitting the events per newline or when it = reaches a fixed size. 
=0A=0A=0A

Hari


On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <simeon_zaharici= @yahoo.com> wrote:
=0A=0A=0A
=
=0A=0A=0AHello=

=0A<= /div>=0A=0A
No, I did not find any solution = to this
=0A=0A=0A=0A
=0A
Simeon
=0A=0A
On Monday, A= pril 7, 2014 6:09:21 AM, Deepak Subhramanian <deepak.subhramanian@gmail.com> wrote:
=0A=0A=0A
Hi Sime= on,
I am facing the same problem. Did you find any solution ?

Thanks, Deepak 


=0A=0A=0A
On Wed, Mar 5, 2014 at 5:= 52 PM, Simeon Zaharici <simeon_zaharici@yahoo.com> wrote:
=0A=0A=0A=0A
=0A=0A=0A=0AHi guys

I wo= uld like to send json data to a flume netcat source and then send it to an = elasticsearch sink that uses the
=0A=0A=0A=0A
org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEve=
ntSerializer.

=0A=0A=0AIt works=
 to send plain text on the netcat source, however when I send json the even=
t ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XCo=
ntentBuilder'.
=0A
Which would be the e=
asiest way to go about this ? I saw from reading docs that creating a morph=
line interceptor could be the way to go but I did not fully understand how =
that=0A works.

Thanks a lot



--
Dee= pak Subhramanian=0A


=


=0A=0A

--1230856681-210918173-1397142786=:99338--