Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
Received-SPF: pass (athena.apache.org: domain of robbie.gemmell@gmail.com
 designates 209.85.161.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEqX17NKATJtOpsJfT2C5cdqStNDZRCJL=yLywqtJ03K6iZV=w@mail.gmail.com>
References: 
 <CAEqX17OJ24AiHuy_V3rkxbLGSw2gXxhovFAWjcqBey1S8PXymA@mail.gmail.com>
	<CAFitrpQTMsWJnwQWKWbcLzR+v9rt1xbXYLJ3MOWZe7TuMaiH4w@mail.gmail.com>
	<CAEqX17NKATJtOpsJfT2C5cdqStNDZRCJL=yLywqtJ03K6iZV=w@mail.gmail.com>
Date: Thu, 24 May 2012 17:45:55 +0100
Message-ID: 
 <CAFitrpRio6F0Kgmx9kqUDxqPM1XRf3WUZKvbEHgCBwRdRgAB4Q@mail.gmail.com>
Subject: Re: Qpid Java Broker Priority Queue Issue!
From: Robbie Gemmell <robbie.gemmell@gmail.com>
To: users@qpid.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Praveen,

I don't believe priority queues are that widely used, but they are
used. The problem with both of these defects is that its quite
possible to never notice/hit them them depending on your messaging
patterns. The newer defect is unfortunate given it appears to require
a series of events to occur and looks to have been introduced while
fixing the other issue before the release, but at first glance it
appears straight forward to remedy and I'll pick that up tomorrow.

Robbie

On 21 May 2012 16:41, Praveen M <lefthandmagic@gmail.com> wrote:
> Hi Robbie,
>
> Thanks a lot for the quick update. Boy, you work late :)
>
> I've raised a Jira for this,
> https://issues.apache.org/jira/browse/QPID-4012?focusedCommentId=3D132802=
20#comment-13280220
>
> My guess is this would impact anyone using priority queues. It is
> unfortunately a show stopper for us :( We did another test run and it
> didn't hold up for over half an hour and the broker ran into this issue.
>
> I'd be great if you can help with this.
>
> Also, I'm curious if priority queues are not extensively used with Qpid.
> (since we ran into two critical issues). Do you know if anyone has
> currently deployed priority queues in production scale?
>
>
> Thanks a lot for helping out.
> Praveen
>
>
>
> On Fri, May 18, 2012 at 7:47 PM, Robbie Gemmell <robbie.gemmell@gmail.com=
>wrote:
>
>> Hi Praveen,
>>
>> Based on a quick look this does appear directly related to the change
>> for the other issue yes, and I think I can guess what probably
>> happened (a previously released entry was compared to a newly released
>> entry after the original entry was consumed, its data removed from the
>> store, and its soft referenced in-memory metadata was garbage
>> collected, so that when inspecting the priority as done for the
>> previous fix we ended up accessing the store again to retrieve
>> metadata later than we can/should). If so its potentially not going to
>> be that consistently reproducable, but has a couple of simple fixes
>> available.
>>
>> I'm busy and/or away until later in the week but if you raise a JIRA
>> then I will pick it up.
>>
>> Probably time to go to sleep before the alarm goes off to wake me up :)
>>
>> Robbie
>>
>> On 19 May 2012 02:17, Praveen M <lefthandmagic@gmail.com> wrote:
>> > Hi,
>> >
>> > We encountered this exception when running some long tests on Qpid usi=
ng
>> > the Java Broker.
>> >
>> > The test uses priority queues with 10 levels.
>> >
>> > I'm curious is this was due to some gaps in the change to fix the bug
>> that
>> > I filed earlier.
>> > https://issues.apache.org/jira/browse/QPID-3927
>> >
>> > Robbie, Can you please let us know if you can think the cause of the
>> > exception below was due to some obvious bug?
>> > I am going to try and reproduce this and provide a test which could
>> > reproduce this soon. Just posting it as a heads up in case you know if
>> > something is amiss atop your head.
>> >
>> >
>> > The exception is,
>> >
>> > qpidbrkr`20120518072800.628`Uncaught exception in thread "IoReceiver -=
 /
>> > 10.224.75.24:34340"
>> >
>> > java.lang.RuntimeException: org.apache.qpid.AMQStoreException: Metadat=
a
>> not
>> > found for message with id 1330312 [error code 541: internal error]
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.store.berkeleydb.BDBMessageStore$StoredBDBMessage=
.getMetaData(BDBMessageStore.java:2327)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.message.MessageTransferMessage.getMetaData(Messag=
eTransferMessage.java:43)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.message.MessageTransferMessage.getMessageHeader(M=
essageTransferMessage.java:58)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.queue.QueueEntryImpl.getMessageHeader(QueueEntryI=
mpl.java:303)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.queue.PriorityQueueList$PriorityQueueEntryImpl.co=
mpareTo(PriorityQueueList.java:189)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.queue.PriorityQueueList$PriorityQueueEntryImpl.co=
mpareTo(PriorityQueueList.java:179)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.queue.SimpleAMQQueue.updateSubRequeueEntry(Simple=
AMQQueue.java:853)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.queue.SimpleAMQQueue.requeue(SimpleAMQQueue.java:=
875)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.queue.QueueEntryImpl.release(QueueEntryImpl.java:=
245)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.subscription.Subscription_0_10.release(Subscripti=
on_0_10.java:715)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.subscription.ExplicitAcceptDispositionChangeListe=
ner.onRelease(ExplicitAcceptDispositionChangeListener.java:62)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.transport.ServerSession.onClose(ServerSession.jav=
a:373)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.transport.ServerSessionDelegate.closed(ServerSess=
ionDelegate.java:1579)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> > org.apache.qpid.transport.Session.closed(Session.java:1097)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> > org.apache.qpid.transport.Connection.closed(Connection.java:541)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.transport.ServerConnection.closed(ServerConnectio=
n.java:420)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> > org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:=
204)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.protocol.ProtocolEngine_0_10.closed(ProtocolEngin=
e_0_10.java:206)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.closed(MultiV=
ersionProtocolEngine.java:102)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> > org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:17=
4)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619=
)
>> >
>> > Caused by: org.apache.qpid.AMQStoreException: Metadata not found for
>> > message with id 1330312 [error code 541: internal error]
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.store.berkeleydb.BDBMessageStore.getMessageMetaDa=
ta(BDBMessageStore.java:1842)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0at
>> >
>> org.apache.qpid.server.store.berkeleydb.BDBMessageStore$StoredBDBMessage=
.getMetaData(BDBMessageStore.java:2323)
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0... 21 more
>> >
>> >
>> > Thank you,
>> >
>> > --
>> > -Praveen
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>
>
> --
> -Praveen

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org