flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Json over netcat source
Date Sun, 13 Apr 2014 14:40:00 GMT
Have been able to reproduce the problem locally using the existing test
cases inside ES Sink. The problem does exist.

Did some initial investigation, the framework is able to detect the JSON
content and tries to add it as complex field.
timestamp is added only if present in header.

In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil

public static void addComplexField(XContentBuilder builder, String
fieldName,
      XContentType contentType, byte[] data) throws IOException {
    XContentParser parser = null;
    try {
      XContentBuilder tmp = jsonBuilder();
      parser = XContentFactory.xContent(contentType).createParser(data);
      parser.nextToken();
      tmp.copyCurrentStructure(parser);
      builder.field(fieldName, tmp); <<<< This is where the we might have
an issue (real action is happening inside this method
                                      call)

Can someone familiar with this part look further into this? I shall debug
further as soon as I have free cycles.

thanks
ashish



On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:

>  Thanks Simon. I am also struggling with no luck. I tried using the latest
> flume elastic search sink jar  build from 1.5SNAPSHOT ,but still no luck. I
> will try to see if it is an issue with elastic search api . When I loaded
> json using hive it loaded JSON properly. But we have to pass a property
> es.input.json in hive.  Is there a way to pass the same in Flume.
>
> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = '...',
>
>               'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>
>


-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message