qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramayan Tiwari <ramayan.tiw...@gmail.com>
Subject Re: org.apache.qpid.server.store.StoreException
Date Thu, 06 Oct 2016 21:33:15 GMT
Hi Rob,

Thanks so much a prompt response and create JIRA to track this.

We don't set a TTL on our messages though -- i.e., we use 0 (unlimited). We
do send it in non-persistent mode; not sure if that matters or could
trigger the same bug?

For the fix, would it be possible to fix it in 0.32 as well as in 6.0.x ?

CCing, Helen from our team.

Thanks,

Ramayan

On Thu, Oct 6, 2016 at 1:21 PM, Rob Godfrey <rob.j.godfrey@gmail.com> wrote:

> I've added a patch file which should apply to 0.32 on the following
> JIRA: https://issues.apache.org/jira/browse/QPID-7451
>
> Hope this helps,
> Rob
>
> On 6 October 2016 at 22:04, Rob Godfrey <rob.j.godfrey@gmail.com> wrote:
> > Hi Ramayan,
> >
> > this is exception indicates that a message has been deleted from the
> > store, but the in-memory queue still references it.  Unfortunately the
> > exception doesn't really tell us anything about how the broker will
> > have got to this state.
> >
> > Having looked at the code I have an idea about what may be happening -
> > do your messages have a TTL set?  I *think* that the AMQP 0-10 message
> > path may be vulnerable to a race condition if a message expires at
> > precisely the same time that the message is picked up to be sent to a
> > consumer (essentially the message has to be available to consume and
> > then three lines of code later it must have been deleted).  The 0-9-1
> > codepath is not vulnerable to this as it caches the size of the
> > message (as you can see in the stack trace, the consumer is checking
> > the size of the message to make sure that the consumer has enough
> > credit to receive it).
> >
> > We're unlikely to do a patch release for 0.32, but we will likely be
> > putting out a new 6.0.x release soon, and soon after a 6.1 release.
> > Would you be able to upgrade to one of these, or would you prefer me
> > to send you a patch file that you could apply to the 0.32 source to
> > test?
> >
> > -- Rob
> >
> > On 6 October 2016 at 21:27, Ramayan Tiwari <ramayan.tiwari@gmail.com>
> wrote:
> >> Hi,
> >>
> >>
> >> We are ran into this StoreException in our production environment
> multiple
> >> times on different brokers, which caused broker shutdown. We are running
> >> 0.32 Java broker with 0.16 client. I see that this was reported and
> fixed
> >> here:
> >> https://issues.apache.org/jira/browse/QPID-4012
> >>
> >> This is still happening, I don't have enough context to reproduce this
> >> locally. Any help is appreciated!
> >>
> >> Thanks
> >> Ramayan
> >>
> >> *Exception*
> >>
> >> Uncaught exception, shutting down.
> >> org.apache.qpid.server.store.StoreException: Metadata not found for
> message
> >> with id 1762118451
> >> at
> >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.
> getMessageMetaData(AbstractBDBMessageStore.java:343)
> >> at
> >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$
> StoredBDBMessage.getMetaData(AbstractBDBMessageStore.java:1224)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.
> getMetaData(MessageTransferMessage.java:41)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.getSize(
> MessageTransferMessage.java:56)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ConsumerTarget_
> 0_10.allocateCredit(ConsumerTarget_0_10.java:494)
> >> at
> >> org.apache.qpid.server.queue.QueueConsumerImpl.wouldSuspend(
> QueueConsumerImpl.java:278)
> >> at
> >> org.apache.qpid.server.queue.AbstractQueue.attemptDelivery(
> AbstractQueue.java:2059)
> >> at
> >> org.apache.qpid.server.queue.AbstractQueue.flushConsumer(
> AbstractQueue.java:1981)
> >> at
> >> org.apache.qpid.server.queue.AbstractQueue.flushConsumer(
> AbstractQueue.java:1957)
> >> at
> >> org.apache.qpid.server.queue.QueueConsumerImpl.flush(
> QueueConsumerImpl.java:318)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ConsumerTarget_
> 0_10.flush(ConsumerTarget_0_10.java:605)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.
> messageFlush(ServerSessionDelegate.java:521)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.
> messageFlush(ServerSessionDelegate.java:82)
> >> at org.apache.qpid.transport.MessageFlush.dispatch(
> MessageFlush.java:87)
> >> at
> >> org.apache.qpid.transport.SessionDelegate.command(
> SessionDelegate.java:55)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(
> ServerSessionDelegate.java:99)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(
> ServerSessionDelegate.java:82)
> >> at org.apache.qpid.transport.Method.delegate(Method.java:159)
> >> at org.apache.qpid.transport.Session.received(Session.java:596)
> >> at org.apache.qpid.transport.Connection.dispatch(Connection.java:452)
> >> at
> >> org.apache.qpid.transport.ConnectionDelegate.handle(
> ConnectionDelegate.java:64)
> >> at
> >> org.apache.qpid.transport.ConnectionDelegate.handle(
> ConnectionDelegate.java:40)
> >> at
> >> org.apache.qpid.transport.MethodDelegate.messageFlush(
> MethodDelegate.java:143)
> >> at org.apache.qpid.transport.MessageFlush.dispatch(
> MessageFlush.java:87)
> >> at
> >> org.apache.qpid.transport.ConnectionDelegate.command(
> ConnectionDelegate.java:54)
> >> at
> >> org.apache.qpid.transport.ConnectionDelegate.command(
> ConnectionDelegate.java:40)
> >> at org.apache.qpid.transport.Method.delegate(Method.java:159)
> >> at org.apache.qpid.transport.Connection.received(Connection.java:405)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerConnection.access$001(
> ServerConnection.java:64)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run(
> ServerConnection.java:316)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run(
> ServerConnection.java:312)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:360)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerConnection.received(
> ServerConnection.java:311)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ServerConnection.received(
> ServerConnection.java:64)
> >> at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97)
> >> at org.apache.qpid.transport.network.Assembler.assemble(
> Assembler.java:198)
> >> at org.apache.qpid.transport.network.Assembler.frame(
> Assembler.java:131)
> >> at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128)
> >> at org.apache.qpid.transport.network.Assembler.received(
> Assembler.java:102)
> >> at org.apache.qpid.transport.network.Assembler.received(
> Assembler.java:44)
> >> at
> >> org.apache.qpid.transport.network.InputHandler.next(
> InputHandler.java:199)
> >> at
> >> org.apache.qpid.transport.network.InputHandler.received(
> InputHandler.java:114)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ProtocolEngine_
> 0_10.received(ProtocolEngine_0_10.java:179)
> >> at
> >> org.apache.qpid.server.protocol.v0_10.ProtocolEngine_
> 0_10.received(ProtocolEngine_0_10.java:43)
> >> at
> >> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(
> MultiVersionProtocolEngine.java:153)
> >> at
> >> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(
> MultiVersionProtocolEngine.java:51)
> >> at org.apache.qpid.transport.network.io.IoReceiver.run(
> IoReceiver.java:161)
> >> at java.lang.Thread.run(Thread.java:745)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message