Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C006F10F67 for ; Thu, 10 Apr 2014 06:48:49 +0000 (UTC) Received: (qmail 48114 invoked by uid 500); 10 Apr 2014 06:48:48 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 47830 invoked by uid 500); 10 Apr 2014 06:48:47 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 47820 invoked by uid 99); 10 Apr 2014 06:48:45 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 06:48:45 +0000 Received: from localhost (HELO [10.0.0.5]) (127.0.0.1) (smtp-auth username mpercy, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 06:48:44 +0000 References: <1394041976.89976.YahooMailNeo@web140802.mail.bf1.yahoo.com> <1397091497.40185.YahooMailNeo@web140803.mail.bf1.yahoo.com> <1397092679.57238.YahooMailNeo@web140803.mail.bf1.yahoo.com> Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: multipart/alternative; boundary=Apple-Mail-06B7CBBB-2B10-4C17-ACC9-BCECE61AB305 Content-Transfer-Encoding: 7bit Message-Id: <49FFBB76-D820-4F8F-BAA4-5DD8420C2B20@apache.org> Cc: "user@flume.apache.org" , Simeon Zaharici X-Mailer: iPhone Mail (11B651) From: Mike Percy Subject: Re: Json over netcat source Date: Wed, 9 Apr 2014 23:48:42 -0700 To: "user@flume.apache.org" --Apple-Mail-06B7CBBB-2B10-4C17-ACC9-BCECE61AB305 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Not sure either but make sure you're using a compatible version of ElasticSe= arch. Sent from my iPhone > On Apr 9, 2014, at 9:43 PM, Hari Shreedharan w= rote: >=20 > Then I really don't know what the issue is. Someone more familiar with ela= stic search sink will need to look at it. >=20 > Hari >=20 >> On Wednesday, April 9, 2014, Simeon Zaharici w= rote: >> Hello >>=20 >> This is to be able to easily integrate log centralization clients such as= nxlog with flume=20 >>=20 >> The problem is not related to the splitting, or the size, even a message l= ike { "blah": "blah" } will result in the exception "org.elasticsearch.commo= n.xcontent.XContentBuilder" >>=20 >>=20 >> Simeon >> On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan wrote: >> Why exactly do you want to send JSON over netcat? Why don't you simply us= e the Avro Source with Flume SDK? I am not entirely sure about the Elastic s= earch sink. The reason your JSON is not working is likely because netcat is s= plitting the events per newline or when it reaches a fixed size.=20 >>=20 >>=20 >> Hari >>=20 >>=20 >> On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici wrote: >> Hello >>=20 >> No, I did not find any solution to this >>=20 >> Simeon >> On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian wrote: >> Hi Simeon, >> I am facing the same problem. Did you find any solution ? >>=20 >> Thanks, Deepak=20 >>=20 >>=20 >> On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici wrote: >> Hi guys >>=20 >> I would like to send json data to a flume netcat source and then send it t= o an elasticsearch sink that uses the >> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.= >>=20 >>=20 >>=20 >>=20 >> It works to send plain text on the netcat source, however when I send jso= n the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.X= ContentBuilder'. >>=20 >>=20 >> Which would be the easiest way to go about this ? I saw from reading docs= that creating a morphline interceptor could be the way to go but I did not f= ully understand how that >> works. >>=20 >> Thanks a lot >>=20 >>=20 >>=20 >> --=20 >> Deepak Subhramanian --Apple-Mail-06B7CBBB-2B10-4C17-ACC9-BCECE61AB305 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
Not sure either but make sure you're using a compatible version of ElasticSearch.

Sent from my iPhone

On Apr 9, 2014, at 9:43 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:

Then I really don't know what the issue is. Someone more familiar with elastic search sink will need to look at it.

Hari

On Wednesday, April 9, 2014, Simeon Zaharici <simeon_zaharici@yahoo.com> wrote:
Hello

This is to be able to easily integrate log centralization clients such as nxlog with flume 

The problem is not related to the splitting, or the size, even a message like { "blah": "blah" } will result in the exception "org.elasticsearch.common.xcontent.XContentBuilder"

Simeon
On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:
Why exactly do you want to send JSON over netcat? Why don't you simply use the Avro Source with Flume SDK? I am not entirely sure about the Elastic search sink. The reason your JSON is not working is likely because netcat is splitting the events per newline or when it reaches a fixed size. 


Hari


On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <simeon_zaharici@yahoo.com> wrote:
Hello

No, I did not find any solution to this

Simeon
On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <deepak.subhramanian@gmail.com> wrote:
Hi Simeon,
I am facing the same problem. Did you find any solution ?

Thanks, Deepak 


On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <simeon_zaharici@yahoo.com> wrote:
Hi guys

I would like to send json data to a flume netcat source and then send it to an elasticsearch sink that uses the 
org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.




It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XContentBuilder'.


Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
 works.

Thanks a lot



--
Deepak Subhramanian





--Apple-Mail-06B7CBBB-2B10-4C17-ACC9-BCECE61AB305--