flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandesh Hurli <sandip...@gmail.com>
Subject Re: Flume MorphlineInterceptor
Date Thu, 27 Feb 2014 10:03:16 GMT
Thanks for the response Wolfgang Hoschek.

I got this working after setting the value for _attachment_body field and
adding a code snippet to convert it to a byte array in the morphline.conf
file.

 Referred the following discussion thread:

https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/9FS3H-ktVmU

Thanks,
Sandesh




On Wed, Feb 26, 2014 at 3:11 PM, Sandesh Hurli <sandip002@gmail.com> wrote:

> Hi,
>
> I am trying process the  flume event body using the Morphline interceptor,
> grok pattern, and write it to a file using file_roll sink (started with a
> simple setup for testing purpose, later will be using HDFS sink).
> The source and the sink start successfully but the output file remains
> empty. Below are the details :
>
> Source is tailing a sample file containing the word "sample". I append
> more words to the file, still dont see any output in the sink destination
> file.
>
> flume.conf file:
> ---------------------------------------------------------------------------
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = loggerSink
>
> agent.sources.seqGenSrc.type = exec
> agent.sources.seqGenSrc.command = tail -F /home/sandesh/sample.txt
> agent.sources.seqGenSrc.interceptors = morphlineinterceptor
> agent.sources.seqGenSrc.interceptors.morphlineinterceptor.type =
> org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
> agent.sources.seqGenSrc.interceptors.morphlineinterceptor.morphlineFile =
> /usr/local/lib/flume-ng/conf/morphline.conf
> agent.sources.seqGenSrc.interceptors.morphlineinterceptor.morphlineId =
> morphline1
> agent.sources.seqGenSrc.channels = memoryChannel
>
> agent.sinks.loggerSink.type = file_roll
> agent.sinks.loggerSink.sink.directory = /home/sandesh/
> agent.sinks.loggerSink.channel = memoryChannel
>
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 1000
>
>
> --------------------------------------------------------------------------------
>
> morphline.conf file:
>
> --------------------------------------------------------------------------------
> morphlines : [
>  {
>     id : morphline1
>     importCommands : ["com.cloudera.**"]
>     commands : [
>             #Reading the line from Flume event:
>            {
>            readLine {
>               charset : UTF-8
>            }
>      }
>
>     {
>             #Regex - pattern matching and transformation will be taken
> care by below config
>      grok {
>          dictionaryFiles :
> [/home/sandesh/cdk/cdk-morphlines/cdk-morphlines-core/src/test/resources/grok-dictionaries]
>          expressions : {
>           message : """%{WORD:word}"""
>          }
>        }
>      }
>    ]
>  }
> ]
>
> ------------------------------------------------------------------------------
>
> Am I missing something obvious? Please help me if yes..
>
> Regards,
> Sandesh
>
>
>
>
>

Mime
View raw message