flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wolfgang Hoschek <whosc...@cloudera.com>
Subject Re: Flume MorphlineInterceptor
Date Wed, 26 Feb 2014 13:46:30 GMT
Firstly, to print diagnostic information such as the content of records as they pass through
the morphline commands, consider enabling TRACE log level, for example by adding the following
line to your log4j.properties file:

log4j.logger.org.kitesdk.morphline=TRACE

Secondly, is it expected that the *entire* line matches the regex labelled "word"? If not
the grok command will fail, of course.

http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#grok

Wolfgang.

On Feb 26, 2014, at 11:41 AM, Sandesh Hurli wrote:

> Hi,
> 
> I am trying process the  flume event body using the Morphline interceptor, grok pattern,
and write it to a file using file_roll sink (started with a simple setup for testing purpose,
later will be using HDFS sink).
> The source and the sink start successfully but the output file remains empty. Below are
the details :
> 
> Source is tailing a sample file containing the word "sample". I append more words to
the file, still dont see any output in the sink destination file.
> 
> flume.conf file:
> ---------------------------------------------------------------------------
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = loggerSink
> 
> agent.sources.seqGenSrc.type = exec
> agent.sources.seqGenSrc.command = tail -F /home/sandesh/sample.txt
> agent.sources.seqGenSrc.interceptors = morphlineinterceptor
> agent.sources.seqGenSrc.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
> agent.sources.seqGenSrc.interceptors.morphlineinterceptor.morphlineFile = /usr/local/lib/flume-ng/conf/morphline.conf
> agent.sources.seqGenSrc.interceptors.morphlineinterceptor.morphlineId = morphline1
> agent.sources.seqGenSrc.channels = memoryChannel
> 
> agent.sinks.loggerSink.type = file_roll
> agent.sinks.loggerSink.sink.directory = /home/sandesh/
> agent.sinks.loggerSink.channel = memoryChannel
> 
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 1000
> 
> --------------------------------------------------------------------------------
> 
> morphline.conf file:
> --------------------------------------------------------------------------------
> morphlines : [
>  {
>     id : morphline1
>     importCommands : ["com.cloudera.**"]
>     commands : [
>             #Reading the line from Flume event:
>            {
>            readLine {
>               charset : UTF-8
>            }
>      }
> 
>     {
>             #Regex - pattern matching and transformation will be taken care by below
config
>      grok {
>          dictionaryFiles :     [/home/sandesh/cdk/cdk-morphlines/cdk-morphlines-core/src/test/resources/grok-dictionaries]
>          expressions : {
>           message : """%{WORD:word}"""
>          }
>        }
>      }
>    ]
>  }
> ]
> ------------------------------------------------------------------------------
> 
> Am I missing something obvious? Please help me if yes..
> 
> Regards,
> Sandesh
> 
> 
> 
> 


Mime
View raw message