incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: Flume-NG Channel Exception
Date Thu, 26 Apr 2012 08:37:57 GMT
Oh, 60k Events are indeed a lot to much for 20mb max heap size. If I calculate right you need
around 600MB Heap. A memory channel with 20mb heap can handle around 2.3K events / second,
an JDBC around 326. Did you batch the lines or read them sequential? Batching could help you
too (catch 1k events and send out, next 1k and so on)

cheers
- Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

On Apr 26, 2012, at 10:28 AM, shekhar sharma wrote:

> Hello Alex,
> One more bit of information is that out of 60000 records, previously i could process
all the 60000 records, but not Esper is logging only 5000 records..
> 
> 
> Regards,
> Som
> 
> On Thu, Apr 26, 2012 at 1:54 PM, shekhar sharma <shekhar2581@gmail.com> wrote:
> HI Alex
> Everything is working fine, Esper is able to detect the events and can write them to
separate log file.
> Well i have written a python script which generates the log file and i am creating almost
60000 records, and if you can see that my exec source tail command is working on that log
file...
> After the events are detected and processed by Esper, and this error does not comes instantenously,
after some time it is thrown...
> 
> Could it me because of -Xmx20 options ?..because the system is my virtual linux box with
1GB RAM and 20GB HD..
> Can the size of the event can also effect? the reason why i am saying this is:
> 
> Event read by exec source is of SimpleEvent which has 2 member variable MAP for headers
and byte[] for body and in Esper if you want to employ filter "IN" or "LIKE" then it requires
string..so what i did, i have created another member variable of type string to convert the
byte array to string. SO while sending this event to the channel, is this bloating up the
channel capacity?
> 
> Regards,
> Som
> 
> 
> 
> On Thu, Apr 26, 2012 at 1:41 PM, alo alt <wget.null@googlemail.com> wrote:
> Thanks,
> 
> org.apache.flume.sink.esper.EsperSink is availabe in the CLASSPATH and flume can load
them? Or did you write the sink?
> 
> best,
>  Alex
> 
> 
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
> 
> On Apr 26, 2012, at 10:03 AM, shekhar sharma wrote:
> 
> > Hello Alex,
> > My configuration file is as follows:
> >
> > host1.properties file
> >
> > host1.sources = avroSource
> > host1.channels = memoryChannel
> > host1.sinks = esper
> >
> > #avroSource configuration
> >
> > host1.sources.avroSource.type = avro
> > host1.sources.avroSource.bind = localhost
> > host1.sources.avroSource.port = 41414
> > host1.sources.avroSource.channels = memoryChannel
> >
> > #Channels
> >
> > host1.channels.memoryChannel.type = memory
> >
> >
> > #Sinks
> > host1.sinks.esper.type = org.apache.flume.sink.esper.EsperSink
> > host1.sinks.esper.channel = memoryChannel
> >
> > host2.properties file:
> >
> > host2.sources = execSource
> > host2.channels = memoryChannel
> > host2.sinks = avroSink
> >
> > #execSource  configuration
> >
> > host2.sources.execSource.type = exec
> > host2.sources.execSource.command = /usr/bin/tail -F /home/dev/LogScripts/json.csv
> > host2.sources.execSource.channels = memoryChannel
> >
> > #Channels
> >
> > host2.channels.memoryChannel.type = memory
> >
> >
> > #Sinks
> > host2.sinks.avroSink.type = avro
> > host2.sinks.avroSink.hostname=localhost
> > host2.sinks.avroSink.port=41414
> > host2.sinks.avroSink.batch-size = 10
> > host2.sinks.avroSink.runner.type = polling
> > host2.sinks.avroSink.runner.polling.interval = 1
> > host2.sinks.avroSink.channel = memoryChannel
> >
> > Regards,
> > Som Shekhar Sharma
> >
> >
> > On Thu, Apr 26, 2012 at 12:35 PM, shekhar sharma <shekhar2581@gmail.com> wrote:
> > Hello,
> > While using memory channel i am getting the following error, what could be the reason
for this:
> > org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired
Sinks are likely not keeping up with sources, or the buffer size is too tight
> >
> >
> > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:84)
> >         at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> >         at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:178)
> >         at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:267)
> >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >         at java.lang.Thread.run(Thread.java:662)
> >
> >
> > Regards,
> > Som
> >
> 
> 
> 


Mime
View raw message