flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Subhramanian <deepak.subhraman...@gmail.com>
Subject Re: Flume ElasticSearch Sink not indexing JSON data
Date Fri, 11 Apr 2014 11:56:10 GMT
 I tried using the latest flume elastic search sink jar  build from
1.5SNAPSHOT ,but still no luck. I will try to see if it is an issue with
elastic search api . When I loaded json data using hive it loaded JSON
properly. But we have to pass a property es.input.json in hive.  Is there a
way to pass a property in Flume.There is another user also posted the same
issue. Any inputs ?


On Wed, Apr 9, 2014 at 2:18 PM, Ashish <paliwalashish@gmail.com> wrote:

> IMHO, checkout the trunk, build it and then just use the agent config from
> old setup. Keep the old setup as is.
>
> thanks
> ashish
>
>
> On Wed, Apr 9, 2014 at 6:17 PM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>> Thanks Otis. I will give it a try. Do I have to replace the flume jars or
>> add the flume jar to the plugins directory to override the current flume
>> version.
>>
>>
>> On Wed, Apr 9, 2014 at 2:19 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> The ES Sink has been modified quite a bit in the trunk.  Would it make
>>> sense to give that a try?
>>>
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>> On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
>>> deepak.subhramanian@gmail.com> wrote:
>>>
>>>> I am posting JSON data to ElasticSearch using Flume . But for some
>>>> reason it is not loading the correct data in the message. Instead of JSON
>>>> message it is storing message data as
>>>> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>>>>
>>>> I tried using the DynamicSerializer which determines the content-type.
>>>> But no luck.
>>>>
>>>> Here is the sample data posted on Elastic Search.
>>>>
>>>> {
>>>>
>>>>   "_index": "test_flume-2014-04-07",
>>>>
>>>>   "_type": "logs",
>>>>
>>>>   "_id": "M9E-33RQTy2kA6QhW6mSUw",
>>>>
>>>>   "_score": null,
>>>>
>>>>   "_source": {
>>>>
>>>>     "@message": "
>>>> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>>>>
>>>>     "@timestamp": "2014-04-07T09:49:26.490Z",
>>>>
>>>>     "@fields": {
>>>>
>>>>       "timestamp": "1396864166490"
>>>>
>>>>     }
>>>>
>>>>   },
>>>>
>>>>   "sort": [
>>>>
>>>>     1396864166490,
>>>>
>>>>     1396864166490
>>>>
>>>>   ]
>>>>
>>>> }
>>>>
>>>> org.elasticsearch.common.xcontent.XContentBuilder
>>>> --
>>>> Deepak Subhramanian
>>>>
>>>
>>>
>>
>>
>> --
>> Deepak Subhramanian
>>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>



-- 
Deepak Subhramanian

Mime
View raw message