flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: Memory Channel Error on Load Testing
Date Fri, 18 Oct 2013 22:55:11 GMT
the error indicates that source is pumping in data faster than the sink is
draining the memory channel. causing the channel to fill up. it does not
appear to be a MemCh issue.


On Fri, Oct 18, 2013 at 8:47 AM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:

> Hi
> I am getting error while load testing a flume agent with HTTSrc and Avro
> Sink.
>
> Is there an optimum configuration for memory channel for get flume working
> on high load. Thanks in advance for any inputs.
>
>
> tier1.sources  = httpsrc1
> tier1.channels = c1   c2
> tier1.sinks    = avrosink1
>
> tier1.sources.httpsrc1.bind     = host1
> tier1.sources.httpsrc1.type = http
> tier1.sources.httpsrc1.port = 443
> tier1.sources.httpsrc1.channels = c1 c2
> tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler
> tier1.sources.httpsrc1.handler.nickname = HTTPTesting
>
> tier1.channels.c1.type   = memory
> tier1.channels.c1.capacity = 10000
> tier1.channels.c1.transactionCapacity = 1000
> tier1.sinks.sink1.channel      = c1
> tier1.sinks.avrosink1.channel      = c1
>
> tier1.channels.c2.type   = memory
> tier1.channels.c2.capacity = 100000
> tier1.channels.c2.transactionCapacity = 10000
>
> #tier1.channels.c1.type   = file
> #tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint
> #tier1.channels.c1.dataDirs = /tmp/flume/data
> #tier1.sinks.avrosink1.channel      = c1
>
>
> tier1.sinks.avrosink1.type = avro
> tier1.sinks.avrosink1.hostname = ipaddr
> tier1.sinks.avrosink1.port = 4444
> tier1.sinks.avrosink1.avro.fileType = DataStream
> tier1.sinks.avrosink1.serializer = avro_event
> tier1.sinks.avrosink1.serializer.compressionCodec = snappy
>
>
>
> tier1.sinks = filesink1
> tier1.sinks.filesink1.type = file_roll
> tier1.sinks.filesink1.channel = c2
> tier1.sinks.filesink1.batchSize = 10000
> #tier1.sinks.filesink1.rollInterval = 86400
> tier1.sinks.filesink1.rollInterval =0
> tier1.sinks.filesink1.sink.directory = /projects/logs
>
>
>
> WARN http.HTTPSource: Error appending event to channel. Channel might be
> full. Consider increasing the channel capacity or make sure the sinks
> perform faster.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: c1}
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
>  at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>  at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>  at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
>  at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>  at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>  at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>  at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.flume.ChannelException: Space for commit to queue
> couldn't be acquired Sinks are likely not keeping up with sources, or the
> buffer size is too tight
>  at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
>
>
> --
> Deepak Subhramanian
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message