qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Mallassi <olivier.malla...@gmail.com>
Subject Re: OutOfMemory exception with a Qpid Java Broker linked to qpid dispatcher
Date Fri, 08 Jul 2016 17:11:16 GMT
We need to run the tests again but you are right. Looking at the Dispatch
Router config, the links are in and out.
thx.


On Fri, Jul 8, 2016 at 5:47 PM, Ted Ross <tross@redhat.com> wrote:

>
>
> On 07/08/2016 11:42 AM, Olivier Mallassi wrote:
>
>> Hi Rob
>>
>> We have the following topology:
>>
>> * Publisher -> Dispatcher -> Broker1 or Broker 2 (Java) -> Dispatcher
>> (same
>> instance) -> Consumer
>> * each Broker is configured (before the run)
>> - with an exchange perf.topic (amqp topic)
>> - with a queue  (perf.queue) that has a binding (binding key + header
>> based
>> filtering) to the perf.topic exchange
>>
>> * the Dispatcher is configured with autoLink
>> * The publisher publish in perf.topic (kind of distributed topic)
>> * The consumer consume from perf.queue (distributed queue)
>> * Dispatch version is 0.6.0 and Broker is in version 6.0.0.
>>
>
> I think your autoLink _in_ from perf.topic is spurious.  I suspect that
> there are no consumers on the router consuming from "perf.topic" and
> therefore the listener attached to the temporary queue is never issued
> credit and it accumulates all perf.topic messages.
>
>
>
>> We have the expected behavior : published messages are load-balanced on
>> brokers and consumed as expected.
>>
>> Yet, and we observed the OOM while running the test for a longer time,
>> when
>> the dispatcher connect, it creates a queue with a UUID (more or less)
>> name.
>> that queue is bind to perf.topic with # as a binding key (so every
>> messages
>> sent by the publisher fill up this queue thus the OOM because it is in
>> memory).
>>
>> It looks to be "expected".
>> Is there a way to avoid this? are we missing something?
>>
>> Many thx.
>>
>> oliv/
>>
>>
>>
>>
>>
>>
>> On Fri, Jul 8, 2016 at 5:16 PM, Rob Godfrey <rob.j.godfrey@gmail.com>
>> wrote:
>>
>> Hi Adel,
>>>
>>> I'm a bit confused... you say
>>>
>>> As no consumer is connected to this queue, all messages are kept
>>>>
>>> in-memory which causes the OutOfMemory exception in the broker.
>>>
>>> but then also say
>>>
>>> Consumer connected to that queue (Is it the dispatcher?):
>>>> Name: qdlink.OyY41QAJRZ4JGGg
>>>> Mode: MOVE
>>>>
>>>
>>> Does the queue have a consumer or not?
>>>
>>> As to holding on to the messages... the heap dump shows that the message
>>> is
>>> in the queue (which we know) and has not been evicted by flow to disk
>>> (we'd
>>> need to work out why)...
>>>
>>> On average what size are the messages you are sending?  Do you see
>>> messages
>>> flow through this topic at all?
>>>
>>> As to the "weird" queue name - what is happening is that we are creating
>>> an
>>> internal temporary queue when you subscribe to an exchange object in the
>>> broker... for that we use a UUID as the queue name.
>>>
>>> -- Rob
>>>
>>> On 8 July 2016 at 15:48, Adel Boutros <adelboutros@live.com> wrote:
>>>
>>> I forgot to add that when I execute a qdmanage command to delete the
>>>> "connector" to the broker, the weird queue disappears.
>>>>
>>>> Adel
>>>>
>>>> Date: Fri, 8 Jul 2016 07:45:49 -0700
>>>>> From: adelboutros@live.com
>>>>> To: users@qpid.apache.org
>>>>> Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to
>>>>>
>>>> qpid dispatcher
>>>>
>>>>>
>>>>> Fixing bad formatting on Nabbledue to Hotmail:
>>>>>
>>>>> Hello,
>>>>>
>>>>> We are doing some performance tests with a Qpid Java Broker connected
>>>>>
>>>> to
>>>
>>>> a
>>>>
>>>>> Qpid dispatcher.
>>>>> We have noticed that after some time, the broker dies with an
>>>>>
>>>> OutOfMemory
>>>
>>>> exception and java heap is dumped.
>>>>> After analyzing the heap dump and checking the broker, we have noticed
>>>>>
>>>> 2
>>>
>>>> weird behaviors:
>>>>>
>>>>> 1) A queue is created on the broker with a weird Name bound on our
>>>>>
>>>> topic
>>>
>>>> (perf.topic) with a binding key "#". It will receive all messages.
>>>>> This queue seems to be created only when the dispatcher is present and
>>>>>
>>>> is
>>>
>>>> configured.
>>>>> As no consumer is connected to this queue, all messages are kept
>>>>>
>>>> in-memory
>>>>
>>>>> which causes the OutOfMemory exception in the broker.
>>>>>
>>>>> Weird Queue definition
>>>>> Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3
>>>>> Type: standard
>>>>> State: ACTIVE
>>>>> Durable: false
>>>>> Lifespan: DELETE_ON_NO_OUTBOUND_LINKS
>>>>> Persist Messages: NEVER
>>>>> Inbound: 0 msg/s (0.00 B/s)
>>>>> Outbound: 0 msg/s (0.00 B/s)
>>>>> Size: 2374 msgs (0.00 B)
>>>>> Pre-fetched: 0 msgs (0.00 B)
>>>>> Oldest Message Age: 356.064 secs
>>>>> Enforced Max. Ttl(ms): 0
>>>>> Enforced Min. Ttl(ms): 0
>>>>> Exclusive: LINK
>>>>>
>>>>> Consumer connected to that queue (Is it the dispatcher?):
>>>>> Name: qdlink.OyY41QAJRZ4JGGg
>>>>> Mode: MOVE
>>>>>
>>>>>
>>>>> 2) All messages even those consumed successfully by the consumers are
>>>>>
>>>> still
>>>>
>>>>> present on the broker
>>>>>
>>>>> From the heap dump, we can see the Berkley DB is keeping a reference
to
>>>>>
>>>> all
>>>>
>>>>> messages. Is this also coming from the above weird queue?
>>>>>
>>>>>
>>>>> PS: If we only use the dispatcher instead, we have none of the weird
>>>>> behaviors
>>>>>
>>>>> Extract from the heap dump (Object holding
>>>>> reference to one of the message header. "Validated" is one the message
>>>>> header fields we set and which is already received by a consumer)
>>>>>
>>>>> char[9] @ 0xf59ff3d8  VALIDATED
>>>>> '- value java.lang.String @ 0xf59ff3c0  VALIDATED
>>>>>   '- value java.util.HashMap$Entry @ 0xf59ff310
>>>>>     '- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0
>>>>>       '- table java.util.HashMap @ 0xf59ff188
>>>>>         '- _appProperties
>>>>> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88
>>>>>           '- _metaData
>>>>>
>>>>>
>>>>
>>> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef
>>>
>>>> @ 0xf59fef70
>>>>>             '- _messageDataRef
>>>>>
>>>>>
>>>>
>>> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage
>>>
>>>> @ 0xf59fef20
>>>>>               '- _handle
>>>>>
>>>> org.apache.qpid.server.protocol.v1_0.Message_1_0 @
>>>>
>>>>> 0xf59feef0
>>>>>                 '- _message
>>>>> org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @
>>>>> 0xf59621c8
>>>>>                   '- _message
>>>>> org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188
>>>>>                     '- _next
>>>>>
>>>> org.apache.qpid.server.queue.StandardQueueEntry
>>>>
>>>>> @ 0xf5962130
>>>>>           - this$0
>>>>>
>>>>>
>>>> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0
>>> @
>>>
>>>> 0xf59ff1f0
>>>>>
>>>>> Broker Config
>>>>> 1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types
>>>>>
>>>> (BDB
>>>>
>>>>> with the default configuration).
>>>>> 1 Topic (perf.topic)
>>>>>      bound to a queue (perfQueue) with a binding key and a jms-selector
>>>>> filter
>>>>>      bound to an "alternate exchange" of a type fanout which is also
>>>>>
>>>> bound
>>>>
>>>>> to a queue but with no binding key (empty string)
>>>>>
>>>>> Dispatcher Config
>>>>>
>>>>> qdmanage -b amqp://localhost:10454 create --type=address
>>>>>
>>>> prefix=perfQueue
>>>
>>>> waypoint=true name=perf.queue.addr
>>>>> qdmanage -b amqp://localhost:10454 create --type=address
>>>>>
>>>> prefix=perf.topic
>>>>
>>>>> waypoint=true name=perf.topic.addr
>>>>> qdmanage -b amqp://localhost:10454 create --type=connector
>>>>> role=route-container addr=localhost port=10455
>>>>> name=localhost.broker.10455.connector
>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink
>>>>>
>>>> addr=perfQueue
>>>
>>>> dir=out connection=localhost.broker.10455.connector
>>>>> name=localhost.broker.10455.perfQueue.out
>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink
>>>>>
>>>> addr=perfQueue
>>>
>>>> dir=in connection=localhost.broker.10455.connector
>>>>> name=localhost.broker.10455.perfQueue.in
>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink
>>>>>
>>>> addr=perf.topic
>>>
>>>> dir=out connection=localhost.broker.10455.connector
>>>>> name=localhost.broker.10455.perf.topic.out
>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink
>>>>>
>>>> addr=perf.topic
>>>
>>>> dir=in connection=localhost.broker.10455.connector
>>>>> name=localhost.broker.10455.perf.topic.in
>>>>>
>>>>> Clients config
>>>>> 3 JMS Consumers connected each to perfQueue on the dispatcher
>>>>> 2 JMS producers connected each to perf.topic on the dispatcher
>>>>>
>>>>> With the above config, we send a number of messages of which only 1/3
>>>>>
>>>> will
>>>>
>>>>> be routed to the "alternate exchange" and never consumed.
>>>>>
>>>>>
>>>>> Versions
>>>>> Qpid Java Broker: 6.0.0
>>>>> Qpid Dispatch: 0.6.0
>>>>> JMS: 0.9.0
>>>>>
>>>>> Regards,
>>>>> Adel
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>>
>>>>
>>>>
>>> http://qpid.2158936.n2.nabble.com/OutOfMemory-exception-with-a-Qpid-Java-Broker-linked-to-qpid-dispatcher-tp7647065p7647066.html
>>>
>>>> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message