storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Radim Kolar (JIRA)" <>
Subject [jira] [Commented] (STORM-339) Severe memory leak to OOM when ackers disabled
Date Fri, 04 Jul 2014 15:40:34 GMT


Radim Kolar commented on STORM-339:

There are 3 methods for implementing protection against OOM without need to acknowledge every
message. Storm in ack mode has 10x lower throughput.

See end of

1) use ring buffer for receiving messages. If messages are processed too slowly newly arriving
message will replace older unprocessed message. This is not a flow control - just protection
against OOM. (type DROP)

2) implement flow control messages, something simple like XON/XOFF protocol (
should suffice (type BLOCK)

3) save messages to disk instead of throwing them away (type PAGE)

for inspiration see

> Severe memory leak to OOM when ackers disabled
> ----------------------------------------------
>                 Key: STORM-339
>                 URL:
>             Project: Apache Storm (Incubating)
>          Issue Type: Bug
>    Affects Versions: 0.9.2-incubating
>            Reporter: Jiahong Li
> Without any ackers enabled, fast component  will continuously leak memory and causing
OOM problems when target component is slow. The OOM problem can be reproduced by running this
> with command:
> {code}
> $ storm jar storm_perf_test-1.0.0-SNAPSHOT-jar-with-dependencies.jar
--spout 1 --bolt 1 --workers 2 --testTime 600 --messageSize 6400
> {code}
> And the worker childopts with {{-Xms2g -Xmx2g -Xmn512m ...}}.
> At the same time, the executed count of target component is far behind from the emitted
count of source component.  I guess it could be that netty client is buffering too much messages
in its message_queue as target component sends back OK/Failure Response too slowly. 

This message was sent by Atlassian JIRA

View raw message