activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikolay Martynov (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMQ-4971) OOM in DemandForwardingBridge
Date Thu, 16 Jan 2014 10:51:23 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873247#comment-13873247
] 

Nikolay Martynov edited comment on AMQ-4971 at 1/16/14 10:51 AM:
-----------------------------------------------------------------

The issue can be easily reproduced in the following scenario:
1. Start 3 brokers. Let it be 0, 1, 2
2. In brokers 1 and 2 enable conduit duplex network connector to 0
3. Connect tcp:// producer for topic A to 0 (jmeter)
4. Connect vm:// consumer for topic A to 1 (simple jms message forwarder in camel: from jms,
to jms)
5. Connect vm:// producer for topic B to 1 (simple jms message forwarder in camel: from jms,
to jms)
6. Connect vm:// consumer for topic B to 2 (simple jms message forwarder in camel: from jms,
to jms)
7. Connect vm:// producer for topic C to 2 (simple jms message forwarder in camel: from jms,
to jms)
8. Connect tcp:// consumer for topic C to 0 (jmeter)
In summary, we just put bi-directional load to each of network bridges where each bridge has
to both send and receive messages and acks.
Since vm:// and camel embedded in the same JVM are faster than tcp:// and peer broker, bridge
very quickly fills heap with responseCallback's used to handle ack and reference message body.
This isnt controlled by broker memory limits or producer flow control and bridges die on OOM.

Additionally, VMTransport will also keep references to asynchronously handled messages. Since
vm:// is faster than tcp://, this is another source of quick OOM where producer flow control
and memory limits have no any effect. While queue is bounded, default size is 2000 and with
1MB messages you quickly run out of heap (doc doesnt seem to explain if and how it could be
adjusted). Wouldnt be a problem if it could be controlled by broker memory limits and producer
flow control.


was (Author: nickolay_martinov):
The issue can be easily reproduced in the following scenario:
1. Start 3 brokers. Let it be 0, 1, 2
2. In brokers 1 and 2 enable conduit duplex network connector to 0
3. Connect tcp:// producer for topic A to 0 (jmeter)
4. Connect vm:// consumer for topic A to 1 (simple jms message forwarder in camel: from jms,
to jms)
5. Connect vm:// producer for topic B to 1 (simple jms message forwarder in camel: from jms,
to jms)
6. Connect vm:// consumer for topic B to 2 (simple jms message forwarder in camel: from jms,
to jms)
7. Connect vm:// producer for topic C to 2 (simple jms message forwarder in camel: from jms,
to jms)
8. Connect tcp:// consumer for topic C to 0 (jmeter)
In summary, we just put bi-directional load to each of network bridges where each bridge has
to both send and receive messages and acks.
Since vm:// and camel embedded in the same JVM are faster than tcp:// and peer broker, bridge
very quickly fills heap with responseCallback's used to handle ack and reference message body.
This isnt controlled by broker memory limits or producer flow control and bridges die on OOM.

Additionally, VMTransport will also keep references to asynchronously handles messages. Since
vm:// is faster than tcp://, this is another source of quick OOM where producer flow control
and memory limits have no any effect. While queue is bounded, default size is 2000 and with
1MB messages you quickly run out of heap (doc doesnt seem to explain if and how it could be
adjusted). Wouldnt be a problem if it could be controlled by broker memory limits and producer
flow control.

> OOM in DemandForwardingBridge
> -----------------------------
>
>                 Key: AMQ-4971
>                 URL: https://issues.apache.org/jira/browse/AMQ-4971
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.0
>            Reporter: Yuriy Sidelnikov
>              Labels: features
>         Attachments: AMQ-4971.patch
>
>
> DemandForwardingBridge sends messages to the other broker and asynchronously waits for
ACKs keeping message bodies in heap. Amount of un-ACK-ed messages kept in heap is not limited.
If local producer is fast then whole heap will be consumed by messages waiting to be ACK-ed
by other broker.
> Possible option to fix the issue:
> Don't wait for ACK from other broker when forwarding the message if some threshold of
un-ACK-ed messages is reached.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message