qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Baronti <f.baro...@list-group.com>
Subject Re: Low throughput with thousands of queues
Date Fri, 15 Nov 2013 16:29:08 GMT
Thanks for your suggestion. I tried to implement it, but I did not get the behaviour I'd like.
If I understood correctly, you suggested to create as many MessageConsumer instances as topics,
and connect every 
consumer to the same queue, each time specifying a different binding key.
What happens in this situation is that, for each message which matches with at least one of
the bindings of the queue, 
one of the consumers is called, in a round-robin fashion. So the first consumer is called
for the first matching 
message, the second consumer for the second message, and so on, with no relationship between
the subject of the message 
and the intended topic of the consumer.
If this is the expected behaviour, then I'm supposed to implement the matching logic in the
application code, which I 
was trying to avoid (in the real application the topics are not always fully-specified, but
often contain # or *).

Flavio


Il 2013/11/15 15:43 PM, Gordon Sim ha scritto:
> On 11/12/2013 04:56 PM, Flavio Baronti wrote:
>> I'm using the C++ broker on Linux, and the
>> Java client on windows.
>> The sender is sending through producers obtained with
>>      MessageProducer topicProducer =
>> session.createProducer(session.createTopic(topicName + "; {node: { type:
>> topic } }"));
>> while the receiver is receiving with a listener set on consumers
>> obtained with
>>      MessageConsumer consumer =
>> session.createConsumer(session.createTopic(topicName));
>> I don't know what to gather from CPU usage. With few sessions it's
>> higher on the broker, and lower on the client.
>> Increasing the number of session it's the other way round, higher on
>> client and lower on broker.
>> In both cases we're very far from 100%.
>>
>> I created a bug report, I'm not too familiar with JIRA so I hope I did
>> it ok.
>> I added the two test programs I'm using. If there is anything else I can
>> help with, please let me know.
>
> Sorry for the delayed response.
>
> The way the c++ broker does delivery is by setting interest in the writability of the
connection if there is a message 
> available. A worker thread will then process that connection which involves attempting
to pull the message off the 
> queue. This gets increasingly expensive as the number of queues from which you are consuming
on that connection goes up.
>
> A suggested change to avoid the problem would be to have all those distinct subscriptions
on the session use the same 
> underlying subscription queue, and just add the bindings for the different keys.
>
> I've attached a patch to your JMSReceiver example that does this. (Unfortunately due
to QPID-5345, the address is 
> somewhat less intuitive than ideal).
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message