activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin C." <mart...@gmx.at>
Subject Re: EOFException when accessing message's properties
Date Sat, 23 Apr 2011 19:56:27 GMT
Hi

for further information if anybody encounters the same problem:

The issue with the corrupted message headers really seems to be
closely related to the persistence, as I got completely rid of the
corrupted messages by disabling parallel dispatching and persistence
in KahaDB by using:

<kahaDB directory="${activemq.base}/data/kahadb"
              checkForCorruptJournalFiles="true" checksumJournalFiles="true"
              concurrentStoreAndDispatchTopics="false"
              concurrentStoreAndDispatchQueues="false"
              />

I am not sure if this really resolves the issue completely, but at
least it didn't happen again for several tenth of millions of messages
until now.

Best regards,
Martin

On Thu, Apr 7, 2011 at 3:15 PM, Martin C. <martinc@gmx.at> wrote:
> Hi,
>
> I investigated the issue further.
>
> In the corresponding message, the "marshalledProperties" seems to be
> corrupt. The length header indicates 4 properties being present, but
> only two marshalled properties are in the "marshalledProperties"
> array. This is the reason for the EOFException during parsing.
>
> Also interestingly, this only happened to persistent messages. On the
> broker, they seem to be valid (they can be browsed in the web
> interface). Until restart of the broker, when they are loaded from
> persistent storage, then I cannot browse for them in the queue due to
> the EOFException happening on the web-interface as well.
>
> Should I search for the issue on the broker, the consumer or the
> producer (my guess being the broker)? Any hints?
>
> Thanks in advance,
> Martin
>
> On Thu, Apr 7, 2011 at 12:19 AM, Martin C. <martinc@gmx.at> wrote:
>> Hi,
>>
>> as a follow-up, I'm currently seeing a very highly elevated error rate
>> on a deployment with 5.4.2. The very strange issue here is that if I
>> move the message out of the DLQ into the original queue, after some
>> attempts it gets consumed without problems.
>>
>> Anybody got any ideas? Could this have to do with prefetch limit being
>> set to 1 and XA transactions (I read about issues with prefetch and
>> spring on the homepage, but this indicates that one should be a valid
>> limit).
>>
>> Thanks in advance for any help!
>>
>> Best regards,
>> Martin
>>
>> ---------- Forwarded message ----------
>> From: Martin C. <martinc@gmx.at>
>> Date: Fri, Mar 11, 2011 at 9:27 AM
>> Subject: EOFException when accessing message's properties
>> To: users <users@activemq.apache.org>
>>
>>
>> Hi,
>>
>> I am experiencing a corrupt message (see stacktrace below) which fails
>> when attempting to access the message's properties. We had this on two
>> occurances until now, both on a 5.3.2 broker. The issue happens
>> persists on 5.4.2, if we start it with the KahaDB datastore of the
>> 5.3.2 instance. The message finally ended in a DLQ, but it also cannot
>> be processed from there.
>>
>> The issue seems to only happen when deserializing the message (either
>> via network or from KahaDB on startup), as the message was viewable in
>> the webinterface of the broker before a restart, but afterwards, the
>> same exception occures when trying to browse the queue containing the
>> message in the web-interface.
>> Unfortunately I do not know a way to reproduce this, as this happens
>> only once every several-million messages. I also unfortunately cannot
>> provide the corrupt datastore due to sensitive data being included.
>>
>> Do you have any idea, if/how I can recover the message somehow?
>>
>> javax.jms.JMSException: java.io.EOFException
>>        at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
>>        at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:199)
>>        at org.apache.activemq.command.ActiveMQMessage.getObjectProperty(ActiveMQMessage.java:462)
>>        at org.apache.activemq.command.ActiveMQMessage.getStringProperty(ActiveMQMessage.java:550)
>>        ... snip: some myapp.internal method calls ...
>>        at myapp.MessageHandler.onMessage(MessageHandler.java:101)
>>        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:534)
>>        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
>>        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
>>        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
>>        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
>>        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.EOFException
>>        at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:572)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:92)
>>        at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:73)
>>        at org.apache.activemq.command.Message.unmarsallProperties(Message.java:189)
>>        at org.apache.activemq.command.Message.getProperty(Message.java:151)
>>        at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:197)
>>        ... 15 more
>>
>>
>> Thanks in advance,
>> Martin
>>
>

Mime
View raw message