flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Czech <eczec...@gmail.com>
Subject Re: Lost Messages in Flume
Date Wed, 25 Feb 2015 19:36:38 GMT
Thanks Jeff!

I misspoke though -- the problem was actually occurring when a *sink* was
connected to a channel that doesn't exist.  In that event, Flume would run
and messages were dequeued from a JMS queue to be seemingly lost in the
channel.  What's the best way to catch that situation and determine some
alternate destination for the messages?

Here's our flume-conf.properties (with some cleansing for
security/readability):

------------
agent.sources = s1
agent.channels = c1 cError
agent.sinks = k1 kError

# Channel configs
agent.channels.c1.type = memory
agent.channels.c1.capacity = 100
agent.channels.cError.type = memory
agent.channels.cError.capacity = 100

# JMS Source
agent.sources.s1.type = jms
agent.sources.s1.channels = c1 c2
agent.sources.s1.initialContextFactory=org.apache.activemq.jndi.ActiveMQInitialContextFactory
agent.sources.s1.providerURL=ssl://localhost:61617
agent.sources.s1.destinationName=MyDestination
agent.sources.s1.destinationType=QUEUE
agent.sources.s1.batchSize=5
agent.sources.s1.sslEnabled=true
agent.sources.s1.messageSelector=ActivityType \= 'MyType1' AND (PayloadType
\= 'Type2' OR PayloadType \= 'Type3')
agent.sources.s1.selector.type=multiplexing
agent.sources.s1.selector.header=PayloadType
agent.sources.s1.selector.mapping.mymapping1=c1  # This channel will end up
not having a sink
agent.sources.s1.selector.default=cError

# Error/default sink
agent.sinks.kError.type = logger
agent.sinks.kError.channel = cError
agent.sinks.kError.type = file_roll
agent.sinks.kError.sink.directory=/Users/me/some/dir1

# Message Sink
agent.sinks.k1.type = logger
agent.sinks.k1.type = file_roll
agent.sinks.k1.sink.directory=/Users/me/some/dir2
agent.sinks.k1.channel = c1xxxx  ### This is where the configuration
mistake is -- this should just be c1 ###
------------

Thanks again!



On Wed, Feb 25, 2015 at 10:48 AM, Jeff Lord <jlord@cloudera.com> wrote:

> Flume will not even start if it is not already started in the context you
> mention. If it is started and you are updating the config than it will
> throw an error in the logs.
>
>
> On Wednesday, February 25, 2015, Eric Czech <eczech52@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I'm trying to figure out what happens in the event of a misconfiguration
>> where a source is connected to a channel that isn't configured.
>>
>> In other words, if channel "my_channel" is connected to an HBase sink
>> (what I want) but I accidentally point my source at "mmmy_channel" in the
>> flume properties file, what happens?  The user guide states that the
>> transaction fails but how can we detect that condition and do something
>> with the event?
>>
>> Some side notes:
>> - We're using a JMS source
>> - We'll definitely use config management or a validation to try to keep
>> misconfigurations from happening, but we still need to know how to deal
>> with when they do
>>
>> Thank you!
>>
>

Mime
View raw message