flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chhaya Vishwakarma <Chhaya.Vishwaka...@lntinfotech.com>
Subject RE: flume agent not starting
Date Thu, 23 Jan 2014 03:08:21 GMT
Hi,

I want multiple line as one flume event 


Regards,
Chhaya Vishwakarma



-----Original Message-----
From: Wolfgang Hoschek [mailto:whoschek@cloudera.com] 
Sent: Wednesday, January 22, 2014 8:00 PM
To: Chhaya Vishwakarma
Cc: user@flume.apache.org
Subject: Re: flume agent not starting

ExecSource always spits out one flume event per line, no matter what command is executed.

On Jan 22, 2014, at 9:09 AM, Chhaya Vishwakarma wrote:

> Hi,
> 
> Is my interceptor configuration correct? Or there is some problem with that ?
> Can someone guide me in this
> Instead of tail I am using CAT now but same error its giving
> 
> Regards,
> Chhaya Vishwakarma
> 
> 
> 
> -----Original Message-----
> From: Chhaya Vishwakarma [mailto:Chhaya.Vishwakarma@lntinfotech.com]
> Sent: Wednesday, January 22, 2014 10:20 AM
> To: user@flume.apache.org
> Subject: RE: flume agent not starting
> 
> Hi ,
> 
> What i should use then for interceptor to work shall I use CAT ??
> 
> -----Original Message-----
> From: Wolfgang Hoschek [mailto:whoschek@cloudera.com]
> Sent: Tuesday, January 21, 2014 6:30 PM
> To: user@flume.apache.org
> Subject: Re: flume agent not starting
> 
> 'tail' with exec source spits out one flume event per line into the interceptor. But
readMultiLine expects multiple lines per event, not one line per event. In other words, by
the time the data arrives in the interceptor it's already too late for readMultiLine to make
sense.
> 
> Wolfgang.
> 
> 
> On Jan 17, 2014, at 2:42 PM, Chhaya Vishwakarma wrote:
> 
>> Hi,
>> 
>> Agent is starting I am using morphline interceptor to parse events
>> 
>> Below is my morphline configuration
>> 
>> morphlines : [
>> {
>> id : morphline1
>> 
>> commands : [
>> {
>> readMultiLine {
>>  regex : "Started session*"
>>  what : next
>>  charset : UTF-8
>> }
>> }
>> ]
>> }
>> ]
>> 
>> This is my flume configuration file
>> 
>> host1.sources = src1
>> host1.sinks = sink1
>> host1.channels = ch1
>> 
>> host1.sources.src1.type = exec
>> host1.sources.src1.command = tail -F
>> /home/user/apache-flume-1.4.0-bin/try.log
>> host1.sources.src1.channels = ch1
>> host1.sources.src1.restart = true
>> #host1.sources.src1.batchsize=10
>> 
>> host1.sources.src1.interceptors = morphlineinterceptor 
>> host1.sources.src1.interceptors.morphlineinterceptor.type = 
>> org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
>> host1.sources.src1.interceptors.morphlineinterceptor.morphlineFile = 
>> /home/user/apache-flume-1.4.0-bin/morphline.conf
>> 
>> host1.sinks.sink1.type = file_roll
>> host1.sinks.sink1.channel = ch1
>> host1.sinks.sink1.sink.directory = /home/user/apache-flume-1.4.0-bin 
>> host1.sinks.sink1.sink.serializer= TEXT
>> host1.sinks.sink1.sink.rollInterval=0
>> 
>> host1.channels.ch1.capacity=1000
>> host1.channels.ch1.transactionCapacity=100
>> 
>> host1.channels.ch1.type=memory
>> 
>> input file content
>> 
>> Started session
>> a
>> b
>> c
>> Started session
>> c
>> d
>> e
>> Started session
>> 
>> But in /home/user/apache-flume-1.4.0-bin I can see a file named
>> 1389960784130-1 but it's not showing any content. I can see only new 
>> line characters in output file
>> 
>> Regards,
>> Chhaya Vishwakarma
>> 
>> 
>> The contents of this e-mail and any attachment(s) may contain confidential or privileged
information for the intended recipient(s). Unintended recipients are prohibited from taking
action on the basis of information in this e-mail and using or disseminating the information,
and must notify the sender and delete it from their system. L&T Infotech will not accept
responsibility or liability for the accuracy or completeness of, or the presence of any virus
or disabling code in this e-mail"
> 


Mime
View raw message