Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
MIME-Version: 1.0
Sender: burtonator2011@gmail.com
In-Reply-To: 
 <CAPVVMa-oL9SX2VczoBgYQKdT8=8grzPk6RNwQLWKNdd-Bru+Ww@mail.gmail.com>
References: 
 <CAAZU44mGjWR-4x68XRwnfk_SkgaT6hk_J1zYu-zYNPxSY=u7Kw@mail.gmail.com>
 <CAPVVMa-oL9SX2VczoBgYQKdT8=8grzPk6RNwQLWKNdd-Bru+Ww@mail.gmail.com>
From: Kevin Burton <burton@spinn3r.com>
Date: Fri, 30 Oct 2015 17:29:21 -0700
Message-ID: 
 <CAAZU44kk+rSHvA386FAPPn3azOCd0RpmLvBP2tdpz+AuhwsSJA@mail.gmail.com>
Subject: Re: Dealing with the "over-prefetch" problem with large numbers of
 workers and many queue servers
To: users@activemq.apache.org
Content-Type: multipart/alternative; boundary=001a11425cf2ce7db005235ba318

--001a11425cf2ce7db005235ba318
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

sorry for the delay in reply.  was dealing with a family issue that I
needed to prioritize...

On Wed, Oct 21, 2015 at 6:52 AM, Tim Bain <tbain@alumni.duke.edu> wrote:

> Right off the top, can't you use INDIVIDUAL_ACK here, rather than
> committing transactions?  That seems like the ideal mode to let you choos=
e
> which messages to ack without having to ack all the ones up to a certain
> point.
>
>
I thought about that. We had moved to sessions to avoid over-indexing
because our tasks create more messages and this way I can bulk commit them
as one unit.

But maybe if I just deal with the "at least once" semantics while the
transactions aren't combined I'll just execute a message at least once.
But there might be a failure scenario where we execute the second message
hundreds of times where if it was a transaction this could be avoided.


>
> Also, I'm curious about how a 30-second message with a prefetch size of 1
> results in a 5-minute latency; why isn't that 2 * 30 seconds =3D 1 minute=
?
>
>
It's because I have one connection per thread per server.

So if we have 10 servers, each thread has ten sessions.  and if prefetch is
1 then that means I prefetch 10 total messages.  If each message takes 30
seconds to execute that thread will take a while to handle all ten.
This leads to significant latency.

I pushed some code last week to instrument this and our average latency
right now is 3-5 minutes between prefetching a message and servicing a
message.

Fortunately there's a timestamp added on prefetch so I can just take the
current time that I am executing the message/task and then subtract the
prefetch time to compute the latency.

Kevin

--=20

We=E2=80=99re hiring if you know of any awesome Java Devops or Linux Operat=
ions
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
=E2=80=A6 or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>

--001a11425cf2ce7db005235ba318--