activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Felix Ehm (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
Date Wed, 22 Jan 2014 11:39:19 GMT

     [ https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Felix Ehm updated AMQ-4986:
---------------------------

    Description: 
We experience this problem in combination with 5.5.1 client and the wireformat.tightEncodingEnabled=false+
failover:

Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g. failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)

Result:
The client closes the connection and re-tries to the broker which in turn throws the following
exception:

{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616] DEBUG Transport
 Transport Connection to: tcp://10.18.3.97:60156 failed: java.io.IOException: Unexpected error
occured: java.lang.OutOfMemoryError: Java heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java heap space
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
	at org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
	at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	... 1 more
{code}


The problem here is that the BaseDataStreamMarshaller reads an int from the buffer and re-uses
it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
        byte rc[] = null;
        if (dataIn.readBoolean()) {
            int size = dataIn.readInt();
            rc = new byte[size];   // PROBLEM! What happens if size has been read and interpreted
wrongly ? 
            dataIn.readFully(rc);
        }
        return rc;
    }
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which triggers the broker
to allocate blindly this amount of mem.

We do not know yet what triggers the wrong byte sequence from the client, but on the brokers
side, there should be a protection against this case.

  was:
We experience this problem in combination with 5.5.1 client and the wireformat.tightEncodingEnabled=true
+ failover:

Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g. failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)

Result:
The client closes the connection and re-tries to the broker which in turn throws the following
exception:

{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616] DEBUG Transport
 Transport Connection to: tcp://10.18.3.97:60156 failed: java.io.IOException: Unexpected error
occured: java.lang.OutOfMemoryError: Java heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java heap space
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
	at org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
	at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	... 1 more
{code}


The problem here is that the BaseDataStreamMarshaller reads an int from the buffer and re-uses
it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
        byte rc[] = null;
        if (dataIn.readBoolean()) {
            int size = dataIn.readInt();
            rc = new byte[size];   // PROBLEM! What happens if size has been read and interpreted
wrongly ? 
            dataIn.readFully(rc);
        }
        return rc;
    }
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which triggers the broker
to allocate blindly this amount of mem.

We do not know yet what triggers the wrong byte sequence from the client, but on the brokers
side, there should be a protection against this case.


> tightEncoding=false + failover triggers Broker memory exhaust
> -------------------------------------------------------------
>
>                 Key: AMQ-4986
>                 URL: https://issues.apache.org/jira/browse/AMQ-4986
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
>         Environment: java 1.7
> Client library 5.5.1
>            Reporter: Felix Ehm
>            Priority: Critical
>             Fix For: 5.10.0
>
>
> We experience this problem in combination with 5.5.1 client and the wireformat.tightEncodingEnabled=false+
failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g. failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn throws the
following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616] DEBUG Transport
 Transport Connection to: tcp://10.18.3.97:60156 failed: java.io.IOException: Unexpected error
occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java heap
space
> 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> 	at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> 	at org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> 	at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> 	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> 	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> 	... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the buffer and
re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
>         byte rc[] = null;
>         if (dataIn.readBoolean()) {
>             int size = dataIn.readInt();
>             rc = new byte[size];   // PROBLEM! What happens if size has been read and
interpreted wrongly ? 
>             dataIn.readFully(rc);
>         }
>         return rc;
>     }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which triggers the
broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but on the
brokers side, there should be a protection against this case.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message