qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adel Boutros <adelbout...@live.com>
Subject RE: [Performance] Benchmarking Qpid dispatch router 0.6.0 with Qpid Java Broker 6.0.0
Date Tue, 02 Aug 2016 18:10:18 GMT
Hello Ted, Gordon,

When I say the JMS producers are sending synchronously, I mean they don't set any options
to the connection URL such as jms.forceAsyncSend. So I guess this means the producer will
wait for the settlement before sending message X + 1.

When I say it fails, I mean that with 1 producer, I have 2500 msg/s. When I add a second producer,
I am at 4800 msg/s (Which is roughly twice the throughput of a single producer). But when
I add a 3rd producer, I am at 5100 msg/s while I except it to be around 7500 msg/s. So for
me the scaling stops working when adding a 3rd producer and above.

What you both explained to me about the single connection is indeed a plausible candidate
because in the tests of "broker only", the throughput of a single connection is around 3 500
msg/s. So on a single connection, I shouldn't go above that figure which is what I am seeing.
I imagine that when I add more producers/consumers, the throughput will shrink even more because
the same connection is used by all the producers and the consumers.

Do you think it might be an a good idea if the connections were per workerThread and not only
a single connection? 

Another solution would be to use a maximum of 3 clients (producer or consumer) per dispatcher
and have a network of interconnected dispatchers but I find it very heavy and hard to maintain
and support on the client-side. Do you agree?

JMS Producer code
ConnectionFactory connectionFactory = new JmsConnectionFactory("amqp://machine:port");
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Topic topic = session.createTopic("perf.topic");
messageProducer = session.createProducer(topic);
messageProducer.send(message);

Regards,
Adel

> Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with Qpid Java Broker
6.0.0
> To: users@qpid.apache.org
> From: tross@redhat.com
> Date: Tue, 2 Aug 2016 13:42:24 -0400
> 
> 
> 
> On 07/29/2016 08:40 AM, Adel Boutros wrote:
> > Hello Ted,
> >
> > Increasing the link capacity had no impact. So, I have
> >  done a series of tests to try and isolate the issue.
> > We tested 3 different architecture without any consumers:
> > Producer --> Broker
> > Producer --> Dispatcher
> > Producer --> Dispatcher --> Broker
> > In every test, we sent 100 000 messages which contained a byte array of 100 bytes.
The producers are sending in synchronous mode and with AUTO_ACKNOWLEDGE.
> >
> > Our benchmark machines have 20 cores and 396 Gb Ram each. We have
> > currently put consumers/producers on 1 machine and dispatcher/brokers on another
machine. They are both connected with a 10 Gbps ethernet connection. Nothing else is using
the machines.
> >
> > The results are in
> >  the table below.
> >
> > What I could observe:
> > The broker alone scales well when I add producers
> > The dispatcher alone scales well when I add producersThe dispatcher connected to
a broker scales well with 2 producersThe dispatcher connected to a broker fails when having
3 producers or more
> 
> In what way does it fail?
> 
> >
> > I
> >  also did some "qdstat -l" while the test was running and at max had 5
> > unsettled deliveries. So I don't think the problem comes with the
> > linkCapacity.
> 
> You mentioned that you are running in synchronous mode.  Does this mean 
> that each producer is waiting for settlement on message X before sending 
> message X+1?
> 
> >
> > What else can we look at? How does the dispatcher connect the producers to the broker?
Does it open a new connection with each new producer? Or does it use some sort of a connection
pool?
> 
> The router multiplexes the broker traffic over a single connection to 
> the broker.
> 
> >
> > Could the issue come from the capacity configuration of the link in the connection
between the broker and the dispatcher?
> 
> Probably not in your case since the backlogs are much smaller than the 
> default capacity.
> 
> >
> >
> >
> >
> >
> >
> >
> >
> >   Number of Producers
> >   Broker
> >   Dispatcher
> >   Combined Producer Throughput (msg/s)
> >   Combined Producer Latency (micros)
> >
> >
> >   1
> >   YES
> >
> >   NO
> >
> >   3 500
> >   370
> >
> >
> >   4
> >   YES
> >   NO
> >
> >   9 200
> >   420
> >
> >
> >   1
> >   NO
> >   YES
> >   6 000
> >   180
> >
> >
> >   2
> >   NO
> >   YES
> >   12 000
> >   192
> >
> >
> >   3
> >   NO
> >   YES
> >   16 000
> >   201
> >
> >
> >   1
> >   YES
> >   YES
> >   2 500
> >   360
> >
> >
> >   2
> >   YES
> >   YES
> >   4 800
> >   400
> >
> >
> >   3
> >   YES
> >   YES
> >   5 200
> >   540
> >
> >
> > qdstat -l
> > bash$ qdstat -b dell445srv:10254 -l
> > Router Links
> >   type      dir  conn id  id  peer  class   addr                  phs  cap  undel
 unsettled  deliveries  admin    oper
> >   =======================================================================================================================
> >   endpoint  in   19       46        mobile  perfQueue             1    250  0  
   0          0           enabled  up
> >   endpoint  out  19       54        mobile  perf.topic              0    250  0
     2          4994922     enabled  up
> >   endpoint  in   27       57        mobile  perf.topic               0    250  0
     1          1678835     enabled  up
> >   endpoint  in   28       58        mobile  perf.topic               0    250  0
     1          1677653     enabled  up
> >   endpoint  in   29       59        mobile  perf.topic               0    250  0
     0          1638434     enabled  up
> >   endpoint  in   47       94        mobile  $management           0    250  0  
   0          1           enabled  up
> >   endpoint  out 47      95        local   temp.2u+DSi+26jT3hvZ       250  0    
 0          0           enabled  up
> >
> > Regards,
> > Adel
> >
> >> Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with Qpid
Java Broker 6.0.0
> >> To: users@qpid.apache.org
> >> From: tross@redhat.com
> >> Date: Tue, 26 Jul 2016 10:32:29 -0400
> >>
> >> Adel,
> >>
> >> That's a good question.  I think it's highly dependent on your
> >> requirements and the environment.  Here are some random thoughts:
> >>
> >>   - There's a trade-off between memory use (message buffering) and
> >>     throughput.  If you have many clients sharing the message bus,
> >>     smaller values of linkCapacity will protect the router memory.  If
> >>     you have relatively few clients wanting to go fast, a larger
> >>     linkCapacity is appropriate.
> >>   - If the underlying network has high latency (satellite links, long
> >>     distances, etc.), larger values of linkCapacity will be needed to
> >>     protect against stalling caused by delayed settlement.
> >>   - The default of 250 is considered a reasonable compromise.  I think a
> >>     value around 10 is better for a shared bus, but 500-1000 might be
> >>     better for throughput with few clients.
> >>
> >> -Ted
> >>
> >>
> >> On 07/26/2016 10:08 AM, Adel Boutros wrote:
> >>> Thanks Ted,
> >>>
> >>> I will try to change linkCapacity. However, I was wondering if there is
a way to "calculate an optimal value for linkCapacity". What factors can impact this field?
> >>>
> >>> Regards,
> >>> Adel
> >>>
> >>>> Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with
Qpid Java Broker 6.0.0
> >>>> To: users@qpid.apache.org
> >>>> From: tross@redhat.com
> >>>> Date: Tue, 26 Jul 2016 09:44:43 -0400
> >>>>
> >>>> Adel,
> >>>>
> >>>> The number of workers should be related to the number of available
> >>>> processor cores, not the volume of work or number of connections.  4
is
> >>>> probably a good number for testing.
> >>>>
> >>>> I'm not sure what the default link credit is for the Java broker (it's
> >>>> 500 for the c++ broker) or the clients you're using.
> >>>>
> >>>> The metric you should adjust is the linkCapacity for the listener and
> >>>> route-container connector.  LinkCapacity is the number of deliveries
> >>>> that can be in-flight (unsettled) on each link.  Qpid Dispatch Router
> >>>> defaults linkCapacity to 250.  Depending on the volumes in your test,
> >>>> this might account for the discrepancy.  You should try increasing this
> >>>> value.
> >>>>
> >>>> Note that linkCapacity is used to set initial credit for your links.
> >>>>
> >>>> -Ted
> >>>>
> >>>> On 07/25/2016 12:10 PM, Adel Boutros wrote:
> >>>>> Hello,We are actually running some performance benchmarks in an
architecture consisting of a Java Broker connected to a Qpid dispatch router. We also have
3 producers and 3 consumers in the test. The producers send message to a topic which has a
binding on a queue with a filter and the consumers receives message from that queue.
> >>>>> We have noticed a significant loss of performance in this architecture
compared to an architecture composed of a simple Java Broker. The throughput of the producers
is down to half and there are a lot of oscillations in the presence of the dispatcher.
> >>>>>
> >>>>> I have tried to double the number of workers on the dispatcher but
it had no impact.
> >>>>>
> >>>>> Can you please help us find the cause of this issue?
> >>>>>
> >>>>> Dispacth router config
> >>>>> router {
> >>>>>     id: router.10454
> >>>>>     mode: interior
> >>>>>     worker-threads: 4
> >>>>> }
> >>>>>
> >>>>> listener {
> >>>>>     host: 0.0.0.0
> >>>>>     port: 10454
> >>>>>     role: normal
> >>>>>     saslMechanisms: ANONYMOUS
> >>>>>     requireSsl: no
> >>>>>     authenticatePeer: no
> >>>>> }
> >>>>>
> >>>>> Java Broker config
> >>>>> export QPID_JAVA_MEM="-Xmx16g -Xms2g"
> >>>>> 1 Topic + 1 Queue
> >>>>> 1 AMQP port without any authentication mechanism (ANONYMOUS)
> >>>>>
> >>>>> Qdmanage on Dispatcher
> >>>>> qdmanage -b amqp://localhost:10454 create --type=address prefix=perfQueue
waypoint=true name=perf.queue.addr
> >>>>> qdmanage -b amqp://localhost:10454 create --type=address prefix=perf.topic
waypoint=true name=perf.topic.addr
> >>>>> qdmanage -b amqp://localhost:10454 create --type=connector role=route-container
addr=localhost port=10455 name=localhost.broker.10455.connector
> >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue
dir=in connection=localhost.broker.10455.connector name=localhost.broker.10455.perfQueue.in
> >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic
dir=out connection=localhost.broker.10455.connector name=localhost.broker.10455.perf.topic.out
> >>>>>
> >>>>> Combined producer throughput
> >>>>> 1 Broker: http://hpics.li/a9d6efa
> >>>>> 1 Broker + 1 Dispatcher: http://hpics.li/189299b
> >>>>>
> >>>>> Regards,
> >>>>> Adel
> >>>>>
> >>>>>  		 	   		
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >>>> For additional commands, e-mail: users-help@qpid.apache.org
> >>>>
> >>>  		 	   		
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> For additional commands, e-mail: users-help@qpid.apache.org
> >>
> >  		 	   		
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message