flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simeon Zaharici <simeon_zahar...@yahoo.com>
Subject Re: Json over netcat source
Date Thu, 10 Apr 2014 15:13:06 GMT
Hello

I followed the instructions here https://flume.apache.org/FlumeUserGuide.html#elasticsearchsink
and I am using a 0.90.7 jar on the flume server against an elastic search cluster running
0.90.7.

Also sending plain text through the netcat source works fine. The only problem is when the
text that is sent is formatted as json

If I send the words plain text over the netcat source it ends up like this in elasticsearch
{"@message":"plain text\r","@timestamp":"2014-04-10T14:16:38.113Z","@fields":{"timestamp":"1397139398113"}}

If I send { “plain” : “text” }

{"@message":"org.elasticsearch.common.xcontent.XContentBuilder@4710ad01","@timestamp":"2014-04-10T14:20:00.871Z","@fields":{"timestamp":"1397139600871"}}

Also sending just {} over the netcat source results in the same error.

Thanks,
Simeon

On Thursday, April 10, 2014 2:48:49 AM, Mike Percy <mpercy@apache.org> wrote:
 
Not sure either but make sure you're using a compatible version of ElasticSearch.

Sent from my iPhone

On Apr 9, 2014, at 9:43 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:


Then I really don't know what the issue is. Someone more familiar with elastic search sink
will need to look at it.
>
>
>Hari
>
>On Wednesday, April 9, 2014, Simeon Zaharici <simeon_zaharici@yahoo.com> wrote:
>
>Hello
>>
>>
>>This is to be able to easily integrate log centralization clients such as nxlog with
flume 
>>
>>
>>The problem is not related to the splitting, or the size, even a message like { "blah":
"blah" } will result in the exception "org.elasticsearch.common.xcontent.XContentBuilder"
>>
>>
>>Simeon
>>On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan <hshreedharan@cloudera.com>
wrote:
>> 
>>Why exactly do you want to send JSON over netcat? Why don't you simply use the Avro
Source with Flume SDK? I am not entirely sure about the Elastic search sink. The reason your
JSON is not working is likely because netcat is splitting the events per newline or when it
reaches a fixed size. 
>>
>>
>>
>>
>>Hari
>>
>>
>>
>>On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <simeon_zaharici@yahoo.com>
wrote:
>>
>>Hello
>>>
>>>
>>>No, I did not find any solution to this
>>>
>>>
>>>Simeon
>>>On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <deepak.subhramanian@gmail.com>
wrote:
>>> 
>>>Hi Simeon,
>>>I am facing the same problem. Did you find any solution ?
>>>
>>>
>>>Thanks, Deepak 
>>>
>>>
>>>
>>>On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <simeon_zaharici@yahoo.com>
wrote:
>>>
>>>Hi guys
>>>>
>>>>
>>>>I would like to send json data to a flume netcat source and then send it to
an elasticsearch sink that uses the 
>>>>org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>>>>
>>>>It works to send plain text on the netcat source, however when I send json
the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XContentBuilder'.
>>>>
>>>>Which would be the easiest way to go about this ? I saw from reading docs
that creating a morphline interceptor could be the way to go but I did not fully understand
how that
 works.
>>>>
>>>>Thanks a lot
>>>>
>>>
>>>
>>>
>>>-- 
>>>Deepak Subhramanian 
>>>
>>>
>>
>>
>>
Mime
View raw message