activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Unexplained slow sending
Date Thu, 29 Aug 2019 12:40:11 GMT
Might the choke point be the NIC on the EC2 instance? If you run the
consumers for A and B on different EC2s, how does that throughput compare
to what you're seeing?

Also, I'd recommend you use JVisualVM or similar to capture a CPU sampling
(not profiling!) snapshot of your producer program to see where it's
spending its time. If there's a significant amount of time spent anywhere
except making the network call to send the bytes of the payload, then dig
into that.

Tim

On Wed, Aug 28, 2019, 12:03 PM alan protasio <alanprot@gmail.com> wrote:

> Hi,
>
> I think you can try disable concurrentStoreAndDispatchQueues and rerun the
> tests.
>
> Alan Diego
>
>
> On Wed, Aug 28, 2019 at 9:42 AM James Green <james.mk.green@gmail.com>
> wrote:
>
> > Hi,
> >
> > Following-up as I've run more tests.
> >
> > My minimal producer suffered the same bug as our main application: we had
> > spring boot activemq thread pooling turned on as a property, but the
> > library (referenced in the main docs) was not included. Looking back at
> my
> > rather sparse notes at the time I activated this my sends of 10K messages
> > went from taking 11m47s to between 2m56s - 3m31s which is a marked
> > improvement.
> >
> > To my chagrin, this has made little different to our real-world
> > application, and so I have modified my minimal producer to be capable of
> > sending to the main application via it's queues.
> >
> > Allow me to elaborate at this point as it's important to understand what
> > I'm looking at...
> >
> > The messages follow a small path through a series of queues as they are
> > processed. Queue A -> B -> C.
> >
> > If my minimal producer sends to Queue C (skipping A and B) I'm able to
> > produce at 49/s which is "quick enough".
> > If my minimal producer sends to Queue B (skipping A) I'm able to produce
> at
> > 28/s - 38/s which is variable but most of the tests reached 38/s.
> > If my minimal producer sends to Queue A I'm able to produce at 28/s -
> 42/s
> > - again variable.
> >
> > Now Queues A and B are consumed by separate Camel routes inside the same
> > application. Queue C is entirely separate.
> >
> > Looking at throughput graphs of the consumption of Queue C, when first
> > going through (A,B) for 10K messages, then going through (B), I can see
> > (A,B) is twice as slow.
> >
> > I'm left wondering if there's contention somehow within the application
> > consuming from (A,B) that is only showing up during load testing on AWS,
> I
> > was not expecting it would be 2x slower unless the producer thread is
> > shared - you might imagine a thread pool was solve that!
> >
> > At this point I have ensured that there are 4 instances of each
> application
> > and they can happily deal with about 50 messages per second across the
> > queues with persistence on. I am uncertain whether I should be expecting
> > more.
> >
> > If anyone has insights on why the two routes within the same application
> > appear contended and indeed on whether overall throughput should be a lot
> > higher I'd love to hear it.
> >
> > James
> >
> >
> > On Thu, 22 Aug 2019 at 14:02, Tim Bain <tbain@alumni.duke.edu> wrote:
> >
> > > Can you create a minimal producer via the OpenWire protocol in Java or
> > > another language of your choice, to determine if your Camel producer is
> > > slow because it's OpenWire or because it's Camel? I suspect you'll find
> > > that OpenWire is the culprit, not Camel, but let's confirm that.
> > >
> > > All of these numbers sound tiny compared to what the ActiveMQ product
> is
> > > capable of (though I don't have any insight into how Amazon has
> > configured
> > > the brokers, nor into any code customizations they might have made). If
> > you
> > > run multiple minimal producers in parallel, does throughput increase
> > > linearly?
> > >
> > > Also, you say you're testing with small payloads; are they small enough
> > > that you might be running into the Nagle algorithm on your TCP sockets?
> > If
> > > you use larger (e.g. 1KB) payloads, what does that do to your
> throughput
> > on
> > > a single producer?
> > >
> > > Tim
> > >
> > > On Thu, Aug 22, 2019, 2:54 AM James Green <james.mk.green@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've been busy shifting an existing workload into AWS recently, and a
> > > load
> > > > test shows a serious performance drop when sending to ActiveMQ which
> I
> > > > could use some advice on.
> > > >
> > > > Quick architecture summary: We send requests via a webserver that are
> > > > forwarded as messages to a queue. A backend receives these messages
> and
> > > > forwards them onward to another queue. Spring Boot with Camel powers
> > the
> > > > show within Docker containers. Messages are persistent.
> > > >
> > > > Story so far:
> > > >
> > > > Tests show this first queue builds rapidly with pending messages yet
> > > > monitoring of our existing production environment shows no such
> > backlog.
> > > >
> > > > Our existing production environment has everything in a single DC so
> > it's
> > > > super low latency. Our AWS environment uses Fargate with AmazonMQ. I
> > > > understand send latency will be higher and AmazonMQ will store the
> > > messages
> > > > across three AZs.
> > > >
> > > > So I launched a small EC2 instance to run some comparison tests:
> > > >
> > > > Receiving via a Camel route is super quick. This is not a problem.
> > > > Sending via a minimal Camel route is super slow. 14 messages per
> > second.
> > > We
> > > > appear to be doing at least 20-30 per second in production but it's
> > > enough
> > > > of a difference.
> > > > Sending via PHP with stomp-php setting both persistence on and
> receipt
> > > > headers on is substantially faster than sending via Camel. 55
> messages
> > > per
> > > > second.
> > > > Tests have been with 10K small payloads.
> > > >
> > > > At this point I'm thinking that both Camel and PHP should be sending
> > with
> > > > the same properties - synchronously and with persistence. The
> messages
> > on
> > > > the queue are flagged persistent when viewed by the web console.
> > > >
> > > > Can anyone provide further suggestions to try?
> > > >
> > > > Thanks,
> > > >
> > > > James
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message