qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Baronti <f.baro...@list-group.com>
Subject Re: Low throughput with thousands of queues
Date Tue, 12 Nov 2013 16:56:06 GMT
Hi Rob,

thanks for your interest. I'm using the C++ broker on Linux, and the Java client on windows.
The sender is sending through producers obtained with
     MessageProducer topicProducer = session.createProducer(session.createTopic(topicName
+ "; {node: { type: topic } }"));
while the receiver is receiving with a listener set on consumers obtained with
     MessageConsumer consumer = session.createConsumer(session.createTopic(topicName));
I don't know what to gather from CPU usage. With few sessions it's higher on the broker, and
lower on the client.
Increasing the number of session it's the other way round, higher on client and lower on broker.
In both cases we're very far from 100%.

I created a bug report, I'm not too familiar with JIRA so I hope I did it ok.
I added the two test programs I'm using. If there is anything else I can help with, please
let me know.

Flavio

Il 2013/11/12 10:36 AM, Rob Godfrey ha scritto:
> Hi Flavio,
>
> a few questions so we can help investigate your issue:
>
> are you using the Java or the C++ broker?
> how are you creating the 100k topics/queues?
> when you run your test is it obvious whether the broker or the client is
> the bottleneck - e.g. is one of these using significantly more CPU time
> than the other?
> would it be possible to provide for you to provide your test code so that
> we can try to replicate the issue (e.g. by attaching to a JIRA)?
>
> thanks,
> Rob
>
>
> On 8 November 2013 12:18, Flavio Baronti <f.baronti@list-group.com> wrote:
>
>> Hello,
>>
>> I have an use case where I need to create hundreds of thousands of queues,
>> each one subscribed to a different topic (therefore I have as many topics
>> as queues).
>> I set up a test with a single producer generating data on a randomly
>> chosen topic, and a receiver retrieving data from the queues (and throwing
>> it away).
>>
>> I'm using the JMS api, and doing the obvious thing makes the throughput
>> drop dramatically from 10k msg/sec with a single topic/queue (around the
>> top my network adapter can sustain) to 20 msg/sec with 100k topics/queues.
>> I found out that I can recover performance by using more JMS sessions and
>> connections - e.g. create 4 connections with 100 sessions each, and
>> randomly distributing the receiving queues on them.
>> This however is less than ideal, since with the JMS client a thread is
>> created for each session, and I definitely don't want 400 threads receiving
>> data. The work I have to do is CPU-bound, and I don't want to waste time in
>> context switching when 2/4 threads can suffice.
>>
>> Why does the throughput drop so badly with many topics/queues? Why adding
>> sessions helps? Am I overlooking something, or doing something wrong?
>>
>> Thanks
>> Flavio
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message