storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jamesweb3@yahoo.com.tw" <jamesw...@yahoo.com.tw>
Subject 回覆: [Issue] OutOfMemoryError when disable ackers
Date Thu, 03 Jul 2014 02:16:05 GMT
Hi,
I really not remove anchor ack when I run storm in the unreliable way by setNumAckers to 0.

I think it's simple by changing config to be between reliable and unreliable storm. I'll try
it later. Thank you

----- Reply message -----
寄件者: "Cody A. Ray" <cody.a.ray@gmail.com>
收件者: <user@storm.incubator.apache.org>
主旨: [Issue] OutOfMemoryError when disable ackers
日期: 週四, 7月 3 日, 2014 年 6:49 上午

My hunch would be that you're anchoring your tuples and without acker tasks these tuple trees
are never destroyed. So all the tuples from each batch are kept in memory until you run out
of memory, i.e., a very, very fast memory leak.


If you disable ackers, you should probably not anchor your tuples either.

-Cody



On Mon, Jun 30, 2014 at 9:56 PM, jamesweb3@yahoo.com.tw <jamesweb3@yahoo.com.tw> wrote:



Hi all,
I use storm 0.9.2 now, and do real-time processing by basic storm. It's normal to use in reliable
way. But the speed is extremely slow about 6000 tuples/s. So I try to disable acker feature
by conf.setNumAckers(0). However, this leads to following error:



java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.reflect.Method.copy(Method.java:151)
at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136) at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
at java.lang.Class.copyMethods(Class.java:2891) at java.lang.Class.getMethods(Class.java:1467)
at clojure.lang.Reflector.getMethods(Reflector.java:357) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:27)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__5748.invoke(worker.clj:128) at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__5483.invoke(executor.clj:256)
at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor..clj:58) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) at backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)




Can anyone tell me why this happens?
Thank you very much.

Best regards,
James Fu







-- 
Cody A. Ray, LEED AP
cody.a.ray@gmail.com
215.501.7891
Mime
View raw message