activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rodos77 <eugene.ro...@nexj.com>
Subject Re: prefetchExtension off-by-1 for transacted consumers with prefetchSize > 0
Date Thu, 15 Apr 2010 15:53:56 GMT

So I finally got around to doing this.  I've made changes on a per
destination basis only for now and have attached the patch to the JIRA.

Also, can you please comment on 
https://issues.apache.org/activemq/browse/AMQ-2659 AMQ-2659 ?  I've proposed
a small code change for that.  Should be a quick fix.  We need this as well
for our stuff to work.

And of course, there are still the issues 
https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652  and 
https://issues.apache.org/activemq/browse/AMQ-2653 AMQ-2653 .  I know those
are much broader and are harder to fix but I just wanted to know if there
are plans.

Thx!


rodos77 wrote:
> 
> Ok, I have to finish some work first but I'll see how much I can get done
> after that.  I'm not too familiar with the classes you mentioned but I'll
> see what I can figure out.  
> 
> In principle, I can wait till 5.4 but I'll see what I can do.
> 
> There is no problem with AMQ RA, it's just that we have our own JMS RA
> that is capable of integrating with any JMS engine among other features. 
> We use this RA in our enterprise stack.
> 
> 
> 
> Gary Tully wrote:
>> 
>> If you are in the thick of it and can propose a patch that would be
>> great.
>> A new boolean option on ActiveMQPolicy would be the best way to introduce
>> the configuration on a destination basis (set in
>> org.apache.activemq.broker.region.policy.PolicyEntry.configure(Broker,
>> SystemUsage, QueueSubscription) and if you want more fine grain control,
>> an
>> option can be added to ConsumerInfo to allow the configuration on a per
>> subscription basis, propagating the change though to a connection and
>> connection factory. That will require regenerating the wireformat
>> marshallers (in activemq-core; mvn -Popenwire-generate generate-sources)
>> so
>> that the new option is propagated from the client. See how you get on.
>> 
>> Otherwise, we can try and get this change into the 5.4 release.
>> 
>> BTW: Is there a problem with the activemq RA that stops you from using
>> it?
>> 
>> 
>> On 22 March 2010 14:56, rodos77 <eugene.rodos@nexj.com> wrote:
>> 
>>>
>>> I would point out that another JIRA that I've opened (
>>> https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652 ),
>>> illustrates
>>> a
>>> deadlock (hang) which is triggered precisely by the prefetch extension
>>> of
>>> transacted consumers (see JIRA for details).  So while trying to avoid
>>> one
>>> potential hang, the use of prefetch extension causes another.
>>>
>>> However, the solution that you propose to make the prefetch extension
>>> configurable and to have it off by default sounds good to me.  How
>>> should
>>> we
>>> proceed to have this put in place?
>>>
>>>
>>>
>>> Gary Tully wrote:
>>> >
>>> > Fair point. From the perspective of the ResourceAdapter where there
>>> are
>>> > explicit expectations this makes good sense. The prefetch should be
>>> > written
>>> > in stone.
>>> >
>>> > From a regular transacted consumers perspective, where the prefetch is
>>> > largely hidden, avoiding a hang for large transactions is good because
>>> a
>>> > hang would be hard to diagnose. I agree it could be considered
>>> user/error
>>> > but it is difficult to flag it as such.
>>> >
>>> > It may make sense to make the (extend the prefetch for a
>>> > transaction) behavior configurable such that it can be disabled for
>>> the
>>> RA
>>> > or off by default and enabled in the event that large transactions
>>> hang.
>>> > Thinking more, off by default is probably the best approach as
>>> > transactions
>>> > that exceed the prefetch should be rare and the RAR behavior would
>>> match
>>> > expectations.
>>> >
>>> > On 18 March 2010 22:05, rodos77 <eugene.rodos@nexj.com> wrote:
>>> >
>>> >>
>>> >> I don't really get this.  If you extend the prefetchSize to more than
>>> >> what
>>> >> it
>>> >> was configured to be by the maxMessages parameter in the
>>> >> Connection.createConnectionConsumer() call, you are violating the JMS
>>> >> spec.
>>> >> The maxMessages parameter is defined as "the maximum number of
>>> messages
>>> >> that
>>> >> can be assigned to a server session at one time", so how can you make
>>> it
>>> >> bigger than what it was set to by the ConnectionConsumer?!
>>> >>
>>> >> ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
>>> >> control the maxMessages argument used in the
>>> >> Connection.createConnectionConsumer() call: maxSessions and
>>> >> maxMessagesPerSessions.  The maxMessages argument is set to their
>>> >> product.
>>> >> This makes sense.  If a user configures ActiveMQ to have x
>>> maxSessions
>>> >> and
>>> >> y
>>> >> maxMessagesPerSessions, maxMessages (and consequentially the
>>> >> prefetchSize)
>>> >> should be set to x*y.  But why should this ever be extended? 
>>> Shouldn't
>>> >> it
>>> >> be expected that only a maximum of x*y messages be delivered at the
>>> same
>>> >> time?  If a user wants to process z <= x*y messages in the same tx,
>>> that
>>> >> should not pose any problem as the RA will have x ServerSessions each
>>> >> capable of processing y messages at the same time.  But if the user
>>> wants
>>> >> to
>>> >> process z > x*y messages at the same time after explicitly
>>> configuring
>>> >> ActiveMQ to only deliver a maximum of x*y messages at the same time,
>>> this
>>> >> is
>>> >> a user error/problem.  ActiveMQ should not be violating the JMS Spec
>>> to
>>> >> accommodate this.  What is the point of having those properties in
>>> the
>>> >> 1st
>>> >> place if you are going to extend them and deliver as many messages as
>>> >> available?
>>> >>
>>> >>
>>> >>
>>> >> Gary Tully wrote:
>>> >> >
>>> >> > The prefetch extension in the transacted case is necessary for
the
>>> case
>>> >> > where a transacted consumer wants to consumer more than prefetch
>>> >> messages
>>> >> > in
>>> >> > a single transaction. Without the extension, the broker would see
a
>>> >> full
>>> >> > prefetch and not dispatch any more and a receive could block.
>>> >> >
>>> >> > On 15 March 2010 21:31, rodos77 <eugene.rodos@nexj.com> wrote:
>>> >> >
>>> >> >>
>>> >> >> JIRA AMQ-2651 opened.
>>> >> >>
>>> >> >> I would be happy to provide a patch but as I stated in my original
>>> >> post,
>>> >> >> I
>>> >> >> don't fully understand the need for prefetchExtension in the
case
>>> of
>>> a
>>> >> >> transacted, asynchronous (prefetchSize > 0) consumer.  If
somebody
>>> >> could
>>> >> >> explain it, that would perhaps give me the required knowledge
to
>>> come
>>> >> up
>>> >> >> with a patch.
>>> >> >>
>>> >> >> The way I see it now, I would get rid of the prefetchExtension
(in
>>> the
>>> >> >> above
>>> >> >> mentioned case) all together, but again I'm probably missing
some
>>> >> piece
>>> >> >> of
>>> >> >> information and I don't want to cause any side effects.  Barring
>>> that,
>>> >> I
>>> >> >> think there is an off-by-1 error in its calculation but I need
>>> this
>>> >> >> confirmed in order to create a proper patch.
>>> >> >> --
>>> >> >> View this message in context:
>>> >> >>
>>> >>
>>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
>>> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>> >> >>
>>> >> >>
>>> >> >
>>> >> >
>>> >> > --
>>> >> > http://blog.garytully.com
>>> >> >
>>> >> > Open Source Integration
>>> >> > http://fusesource.com
>>> >> >
>>> >> >
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > http://blog.garytully.com
>>> >
>>> > Open Source Integration
>>> > http://fusesource.com
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27987505.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
>> -- 
>> http://blog.garytully.com
>> 
>> Open Source Integration
>> http://fusesource.com
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p28257116.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message