activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thiago Kronig <thiagokro...@gmail.com>
Subject Re: Cannot concurrently consume from embedded broker
Date Mon, 02 Feb 2015 23:07:39 GMT
Thank you!

On Mon Feb 02 2015 at 9:06:40 PM Timothy Bish <tabish121@gmail.com> wrote:

>
> On 02/02/2015 05:59 PM, Thiago Kronig wrote:
> > Timothy, last question, I promise.
> >
> > Prefetch = 1 means that at most 2 messages will be dispatched to a
> > consumer, the in-flight message and the "prefetched" one? I ask you
> because
> > setting to one resulted in four consumers with dispatched counter = 1,
> > three consumers with dispatched counter = 2, and the other three
> consumers
> > waiting for messages, as seen via MBean.
> > On Mon Feb 02 2015 at 8:34:48 PM Timothy Bish <tabish121@gmail.com>
> wrote:
> You can reduce the chance by turning of prefetch extension, see:
> http://activemq.apache.org/per-destination-policies.html
>
> >> On 02/02/2015 05:27 PM, Thiago Kronig wrote:
> >>> First of all, thank you Timothy and Tim for your time.
> >>>
> >>> In fact, setting jms.prefetchPolicy.all=0 solves the problem, because
> >>> "*Specifying
> >>> a prefetch limit of zero means the consumer will poll for more
> messages,
> >>> one at a time"*.
> >>>
> >>> Please, validate my understanding: the default prefetch size is 1000
> >>> messages for some queue costumer, which means that all my 10 consumers
> >> were
> >>> actually competing to dequeue 1000 messages from the broker. The
> broker,
> >> in
> >>> some runnings dispatched 2 messages to one and 8 messages to another.
> >> With competing consumers that have prefetch things will eventuall smooth
> >> out to where the broker will round robin dispatch but only after they
> >> are all online, prior to that the broker will dispatch based on the
> >> arrival of the consumers so the first one likely got seven then the
> >> second got one and then first got one...or other variations.
> >>
> >> You could set prefetch to one, but you can still have a slight imbalance
> >> depending on how fast the messages get processed on each consumer.  You
> >> need to determine the optimal configuration based on your use case and
> >> requirements.
> >>
> >>> On Mon Feb 02 2015 at 7:59:26 PM Timothy Bish <tabish121@gmail.com>
> >> wrote:
> >>>> On 02/02/2015 04:47 PM, Thiago Kronig wrote:
> >>>>> I just tried the same approach using JMS consumers. No success.
> >>>>> Source at:
> >>>>> https://github.com/thiagokronig/activemq-camel-
> >>>> test/blob/master/src/main/java/com/thiagokronig/activemqcameltest/
> >>>> ActiveMQJMSParallelTest.java
> >>>>> On Mon Feb 02 2015 at 7:30:00 PM Thiago Kronig <
> thiagokronig@gmail.com
> >>>>> wrote:
> >>>> You need to take consumer prefetch values into account when dealing
> with
> >>>> concurrent consumers.
> >>>> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
> >>>>
> >>>>>> Tim, you're right: 1 second is to small timeout to begin with.
> >>>>>> But even if I set to 10 minutes, I get the same discouraging
> results:
> >> 3
> >>>>>> consumers, sometimes 8 consumers processing different messages
in
> >>>> parallel.
> >>>>>> Sometimes 10 consumers, as my test requires, consumes all the
> messages
> >>>> in
> >>>>>> parallel and exit before the `10 minute` timeout; but that is
> >>>> *sometimes*.
> >>>>>> On Mon Feb 02 2015 at 7:12:58 PM Tim Bain <tbain@alumni.duke.edu>
> >>>> wrote:
> >>>>>>> Are you sure that your assumption that a Camel context can
be
> >>>> initialized
> >>>>>>> enough to consume messages in all 10 threads within 1 second
is a
> >> valid
> >>>>>>> one?  In my experience, Camel contexts aren't quick to start,
and
> it
> >>>>>>> wouldn't surprise me if there was some asynchronous startup
logic
> >> that
> >>>>>>> took
> >>>>>>> place in parallel after control returns from you call to
> >>>>>>> camelContext.addRoutes().  Can you point us to any documentation
> that
> >>>>>>> validates that startup is supposed to work as quickly as
your code
> >>>>>>> assumes?  If not, you could increase the timeouts to 2 seconds,
> and I
> >>>>>>> would
> >>>>>>> expect your test would start succeeding most/all of the
time...
> >>>>>>>
> >>>>>>> On Mon, Feb 2, 2015 at 1:49 PM, Thiago Kronig <
> >> thiagokronig@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Tim, other thing: I tried to Thread.sleep(5000); between
the
> >> producer
> >>>>>>>> commit and the consumer initialization, but had no success.
> >>>>>>>>
> >>>>>>>> On Mon Feb 02 2015 at 6:45:19 PM Thiago Kronig <
> >>>> thiagokronig@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Tim, in may scenario I want to consume N messages
in N consumers,
> >> one
> >>>>>>> for
> >>>>>>>>> each. My consumers wait on a barrier only to show
the
> parallelism,
> >>>>>>> and to
> >>>>>>>>> show that none of them processed two messages or
more. Maybe my
> >> Camel
> >>>>>>>>> configuration is wrong, or there is something wrong
with the
> broker
> >>>>>>>>> configuration.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon Feb 02 2015 at 6:17:38 PM Tim Bain <tbain@alumni.duke.edu
> >
> >>>>>>> wrote:
> >>>>>>>>>> Why do your consumers all wait for the Nth message
to be
> received
> >>>>>>> before
> >>>>>>>>>> they return to process another?  Why don't you
use an
> >> AtomicInteger
> >>>>>>> to
> >>>>>>>>>> subtract one each time, and succeed if you hit
0 and fail if you
> >>>>>>> haven't
> >>>>>>>>>> hit 0 by the end of your timeout interval? 
Based on what you've
> >>>>>>> shows
> >>>>>>>> us
> >>>>>>>>>> so far, this appears to be an artifact of your
test code, not a
> >>>>>>> problem
> >>>>>>>>>> with the broker.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Feb 2, 2015 at 1:02 PM, Thiago Kronig
<
> >>>>>>> thiagokronig@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi list.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm trying to concurrently consume 10 messages
sent to an
> >> embedded
> >>>>>>>>>> ActiveMQ
> >>>>>>>>>>> broker over the VM transport. Sometimes
my code works,
> sometimes
> >> it
> >>>>>>>>>> hangs
> >>>>>>>>>>> at the n-th message, randomly.
> >>>>>>>>>>>
> >>>>>>>>>>> If I change to a JBossMQ, my Camel client
works.
> >>>>>>>>>>>
> >>>>>>>>>>> Can someone help me in setting my ActiveMQ
broker?
> >>>>>>>>>>>
> >>>>>>>>>>> Source at: https://github.com/thiagokronig/activemq-camel-test
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks in advance.
> >>>>>>>>>>>
> >>>> --
> >>>> Tim Bish
> >>>> Sr Software Engineer | RedHat Inc.
> >>>> tim.bish@redhat.com | www.redhat.com
> >>>> skype: tabish121 | twitter: @tabish121
> >>>> blog: http://timbish.blogspot.com/
> >>>>
> >>>>
> >>
> >> --
> >> Tim Bish
> >> Sr Software Engineer | RedHat Inc.
> >> tim.bish@redhat.com | www.redhat.com
> >> skype: tabish121 | twitter: @tabish121
> >> blog: http://timbish.blogspot.com/
> >>
> >>
>
>
> --
> Tim Bish
> Sr Software Engineer | RedHat Inc.
> tim.bish@redhat.com | www.redhat.com
> skype: tabish121 | twitter: @tabish121
> blog: http://timbish.blogspot.com/
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message