qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adel Boutros <adelbout...@live.com>
Subject OutOfMemory exception with a Qpid Java Broker linked to qpid dispatcher
Date Fri, 08 Jul 2016 14:40:53 GMT
Hello,

We are doing some performance tests with a Qpid Java Broker connected to a Qpid dispatcher.
We have noticed that after some time, the broker dies with an OutOfMemory exception and java
heap is dumped.
After analyzing the heap dump and checking the broker, we have noticed 2 weird behaviors:

A queue is created on the broker with a weird Name bound on our topic 
(perf.topic) with a binding key "#"
It will receive all messages. 
This queue seems to be created only when the dispatcher is present and is configured.
As no consumer is connected to this queue, all messages are kept in-memory which causes the
OutOfMemory exception in the broker.

Weird Queue definition
Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3
Type: standard
State: ACTIVE
Durable: false
Lifespan: DELETE_ON_NO_OUTBOUND_LINKS
Persist Messages: NEVER
Inbound: 0 msg/s (0.00 B/s)
Outbound: 0 msg/s (0.00 B/s)
Size: 2374 msgs (0.00 B)
Pre-fetched: 0 msgs (0.00 B)
Oldest Message Age: 356.064 secs
Enforced Max. Ttl(ms): 0
Enforced Min. Ttl(ms): 0
Exclusive: LINK

Consumer connected to that queue (Is it the dispatcher?): 
Name: qdlink.OyY41QAJRZ4JGGg
Mode: MOVE


All messages even those consumed successfully by the consumers are still present on the broker

>From the heap dump, we can see the Berkley DB is keeping a reference to all messages.
Is this also coming from the above weird queue?


PS: If we only use the dispatcher instead, we have none of the weird behaviors



Extract from the heap dump (Object holding 
reference to one of the message header. "Validated" is one the message 
header fields we set and which is already received by a consumer)
char[9] @ 0xf59ff3d8  VALIDATED                                                          
       
'- value java.lang.String @ 0xf59ff3c0  VALIDATED                                        
             
  '- value java.util.HashMap$Entry @ 0xf59ff310                                          
            
    '- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0                                       
            
      '- table java.util.HashMap @ 0xf59ff188                                            
          
        '- _appProperties org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88
                    
          '- _metaData org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef
@ 0xf59fef70     
            '- _messageDataRef org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage
@ 0xf59fef20
              '- _handle org.apache.qpid.server.protocol.v1_0.Message_1_0 @ 0xf59feef0   
                     
                '- _message org.apache.qpid.server.message.AbstractServerMessageImpl$Reference
@ 0xf59621c8          
                  '- _message org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188
                    
                    '- _next org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962130
                    
          - this$0 org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0
@ 0xf59ff1f0    
Broker Config
1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types (BDB with the default
configuration)
1 Topic (perf.topic)
bound to a queue (perfQueue) with a binding key and a jms-selector filterbound to an "alternate
exchange" of a type fanout which is also bound to a queue but with no binding key (empty string)

Dispatcher Config

qdmanage -b amqp://localhost:10454 create --type=address prefix=perfQueue waypoint=true name=perf.queue.addr
qdmanage -b amqp://localhost:10454 create --type=address prefix=perf.topic waypoint=true name=perf.topic.addr
qdmanage -b amqp://localhost:10454 create --type=connector role=route-container addr=localhost
port=10455 name=localhost.broker.10455.connector
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.out
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue dir=in connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.in
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.out
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic dir=in connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.in

Clients config
3 JMS Consumers connected each to perfQueue on the dispatcher2 JMS producers connected each
to perf.topic on the dispatcher

With the above config, we send a number of messages of which only 1/3 will be routed to the
"alternate exchange" and never consumed.


Versions
Qpid Java Broker: 6.0.0Qpid Dispatch: 0.6.0JMS: 0.9.0

Regards,
Adel

 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message