activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: Dealing with the "over-prefetch" problem with large numbers of workers and many queue servers
Date Sat, 31 Oct 2015 00:29:21 GMT
sorry for the delay in reply.  was dealing with a family issue that I
needed to prioritize...

On Wed, Oct 21, 2015 at 6:52 AM, Tim Bain <tbain@alumni.duke.edu> wrote:

> Right off the top, can't you use INDIVIDUAL_ACK here, rather than
> committing transactions?  That seems like the ideal mode to let you choose
> which messages to ack without having to ack all the ones up to a certain
> point.
>
>
I thought about that. We had moved to sessions to avoid over-indexing
because our tasks create more messages and this way I can bulk commit them
as one unit.

But maybe if I just deal with the "at least once" semantics while the
transactions aren't combined I'll just execute a message at least once.
But there might be a failure scenario where we execute the second message
hundreds of times where if it was a transaction this could be avoided.


>
> Also, I'm curious about how a 30-second message with a prefetch size of 1
> results in a 5-minute latency; why isn't that 2 * 30 seconds = 1 minute?
>
>
It's because I have one connection per thread per server.

So if we have 10 servers, each thread has ten sessions.  and if prefetch is
1 then that means I prefetch 10 total messages.  If each message takes 30
seconds to execute that thread will take a while to handle all ten.
This leads to significant latency.

I pushed some code last week to instrument this and our average latency
right now is 3-5 minutes between prefetching a message and servicing a
message.

Fortunately there's a timestamp added on prefetch so I can just take the
current time that I am executing the message/task and then subtract the
prefetch time to compute the latency.

Kevin

-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message