activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thiago Kronig <thiagokro...@gmail.com>
Subject Re: Cannot concurrently consume from embedded broker
Date Mon, 02 Feb 2015 22:59:16 GMT
Timothy, last question, I promise.

Prefetch = 1 means that at most 2 messages will be dispatched to a
consumer, the in-flight message and the "prefetched" one? I ask you because
setting to one resulted in four consumers with dispatched counter = 1,
three consumers with dispatched counter = 2, and the other three consumers
waiting for messages, as seen via MBean.
On Mon Feb 02 2015 at 8:34:48 PM Timothy Bish <tabish121@gmail.com> wrote:

> On 02/02/2015 05:27 PM, Thiago Kronig wrote:
> > First of all, thank you Timothy and Tim for your time.
> >
> > In fact, setting jms.prefetchPolicy.all=0 solves the problem, because
> > "*Specifying
> > a prefetch limit of zero means the consumer will poll for more messages,
> > one at a time"*.
> >
> > Please, validate my understanding: the default prefetch size is 1000
> > messages for some queue costumer, which means that all my 10 consumers
> were
> > actually competing to dequeue 1000 messages from the broker. The broker,
> in
> > some runnings dispatched 2 messages to one and 8 messages to another.
>
> With competing consumers that have prefetch things will eventuall smooth
> out to where the broker will round robin dispatch but only after they
> are all online, prior to that the broker will dispatch based on the
> arrival of the consumers so the first one likely got seven then the
> second got one and then first got one...or other variations.
>
> You could set prefetch to one, but you can still have a slight imbalance
> depending on how fast the messages get processed on each consumer.  You
> need to determine the optimal configuration based on your use case and
> requirements.
>
> >
> > On Mon Feb 02 2015 at 7:59:26 PM Timothy Bish <tabish121@gmail.com>
> wrote:
> >
> >> On 02/02/2015 04:47 PM, Thiago Kronig wrote:
> >>> I just tried the same approach using JMS consumers. No success.
> >>> Source at:
> >>> https://github.com/thiagokronig/activemq-camel-
> >> test/blob/master/src/main/java/com/thiagokronig/activemqcameltest/
> >> ActiveMQJMSParallelTest.java
> >>> On Mon Feb 02 2015 at 7:30:00 PM Thiago Kronig <thiagokronig@gmail.com
> >
> >>> wrote:
> >> You need to take consumer prefetch values into account when dealing with
> >> concurrent consumers.
> >> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
> >>
> >>>> Tim, you're right: 1 second is to small timeout to begin with.
> >>>> But even if I set to 10 minutes, I get the same discouraging results:
> 3
> >>>> consumers, sometimes 8 consumers processing different messages in
> >> parallel.
> >>>> Sometimes 10 consumers, as my test requires, consumes all the messages
> >> in
> >>>> parallel and exit before the `10 minute` timeout; but that is
> >> *sometimes*.
> >>>>
> >>>> On Mon Feb 02 2015 at 7:12:58 PM Tim Bain <tbain@alumni.duke.edu>
> >> wrote:
> >>>>> Are you sure that your assumption that a Camel context can be
> >> initialized
> >>>>> enough to consume messages in all 10 threads within 1 second is
a
> valid
> >>>>> one?  In my experience, Camel contexts aren't quick to start, and
it
> >>>>> wouldn't surprise me if there was some asynchronous startup logic
> that
> >>>>> took
> >>>>> place in parallel after control returns from you call to
> >>>>> camelContext.addRoutes().  Can you point us to any documentation
that
> >>>>> validates that startup is supposed to work as quickly as your code
> >>>>> assumes?  If not, you could increase the timeouts to 2 seconds,
and I
> >>>>> would
> >>>>> expect your test would start succeeding most/all of the time...
> >>>>>
> >>>>> On Mon, Feb 2, 2015 at 1:49 PM, Thiago Kronig <
> thiagokronig@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Tim, other thing: I tried to Thread.sleep(5000); between the
> producer
> >>>>>> commit and the consumer initialization, but had no success.
> >>>>>>
> >>>>>> On Mon Feb 02 2015 at 6:45:19 PM Thiago Kronig <
> >> thiagokronig@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Tim, in may scenario I want to consume N messages in N consumers,
> one
> >>>>> for
> >>>>>>> each. My consumers wait on a barrier only to show the parallelism,
> >>>>> and to
> >>>>>>> show that none of them processed two messages or more. Maybe
my
> Camel
> >>>>>>> configuration is wrong, or there is something wrong with
the broker
> >>>>>>> configuration.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon Feb 02 2015 at 6:17:38 PM Tim Bain <tbain@alumni.duke.edu>
> >>>>> wrote:
> >>>>>>>> Why do your consumers all wait for the Nth message to
be received
> >>>>> before
> >>>>>>>> they return to process another?  Why don't you use an
> AtomicInteger
> >>>>> to
> >>>>>>>> subtract one each time, and succeed if you hit 0 and
fail if you
> >>>>> haven't
> >>>>>>>> hit 0 by the end of your timeout interval?  Based on
what you've
> >>>>> shows
> >>>>>> us
> >>>>>>>> so far, this appears to be an artifact of your test
code, not a
> >>>>> problem
> >>>>>>>> with the broker.
> >>>>>>>>
> >>>>>>>> On Mon, Feb 2, 2015 at 1:02 PM, Thiago Kronig <
> >>>>> thiagokronig@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi list.
> >>>>>>>>>
> >>>>>>>>> I'm trying to concurrently consume 10 messages sent
to an
> embedded
> >>>>>>>> ActiveMQ
> >>>>>>>>> broker over the VM transport. Sometimes my code
works, sometimes
> it
> >>>>>>>> hangs
> >>>>>>>>> at the n-th message, randomly.
> >>>>>>>>>
> >>>>>>>>> If I change to a JBossMQ, my Camel client works.
> >>>>>>>>>
> >>>>>>>>> Can someone help me in setting my ActiveMQ broker?
> >>>>>>>>>
> >>>>>>>>> Source at: https://github.com/thiagokronig/activemq-camel-test
> >>>>>>>>>
> >>>>>>>>> Thanks in advance.
> >>>>>>>>>
> >>
> >> --
> >> Tim Bish
> >> Sr Software Engineer | RedHat Inc.
> >> tim.bish@redhat.com | www.redhat.com
> >> skype: tabish121 | twitter: @tabish121
> >> blog: http://timbish.blogspot.com/
> >>
> >>
>
>
> --
> Tim Bish
> Sr Software Engineer | RedHat Inc.
> tim.bish@redhat.com | www.redhat.com
> skype: tabish121 | twitter: @tabish121
> blog: http://timbish.blogspot.com/
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message