flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject RE: Channel full exception
Date Tue, 21 Apr 2015 18:28:05 GMT
Increase it to a really large value. It is just meant to guard against rogue clients killing
the agent by r forcing to OOM





Thanks, Hari

On Tue, Apr 21, 2015 at 5:12 AM, mahendran m <mahendranec@hotmail.com>
wrote:

> But i don't know the size of list . i can only find the size of list at converter while
converting the message to List<Event> , then how is it possible to increase the transaction
capacity in CONF ?
> Thanks,Mahendran
> Date: Tue, 21 Apr 2015 02:11:33 -0700
> From: hshreedharan@cloudera.com
> To: user@flume.apache.org
> CC: user@flume.apache.org
> Subject: Re: Channel full exception
> You need to increase the transaction capacity to the maximum size the list could be.
> Thanks, Hari
> On Tue, Apr 21, 2015 at 12:29 AM, mahendran m <mahendranec@hotmail.com> wrote:
> Hi , 
> I am using the flume JMS source to dequeue message from ActiveMQ and convert this message
into `List<Event>` using custom converter
> Channel configuration
>     agent.channels.c1.type = memory
>     agent.channels.c1.capacity = 1000000
>     agent.channels.c1.transactionCapacity = 1500
> when size of the `List<Event>` is less than or equal to 1500 (channel transaction
capacity),then flume write the events to channel , but if the event size is greater than 1500
then  i am getting the below exception 
> Error log
>     21 Apr 2015 12:19:28,245 WARN  [PollableSourceRunner-JMSSource-s1] (org.apache.flume.source.jms.JMSSource.doProcess:263)
 - Error appending event to channel. Channel might be full. Consider increasing the channel
capacity or make sure the sinks perform faster.
>     org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name:
c1}
>     	at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>     	at org.apache.flume.source.jms.JMSSource.doProcess(JMSSource.java:257)
>     	at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:54)
>     	at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
>     	at java.lang.Thread.run(Thread.java:722)
>     Caused by: org.apache.flume.ChannelException: Put queue for MemoryTransaction of
capacity 1500 full, consider committing more frequently, increasing capacity or increasing
thread count
>     	at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doPut(MemoryChannel.java:84)
>     	at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>     	at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>     	at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>     	... 4 more
> How to solve this problem? 
>  
> Note: Event size varies dynamically basedon ActiveMQ message 
> Thanks, 		 	   		  
>  		 	   		  
Mime
View raw message