activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Date Wed, 16 Apr 2008 18:15:04 GMT
Hopefully!
On 16 Apr 2008, at 18:39, yaussy wrote:

>
> Rob,
>
> So, I would find this under the 5.2 SNAPSHOT tomorrow morning (USA  
> central
> standard time)?
>
> Kevin
>
>
> rajdavies wrote:
>>
>> yep - being able to reproduce - this is now fixed in trunk - but too
>> late for the 5.1 release
>>
>> thanks,
>>
>> Rob
>>
>> On 16 Apr 2008, at 13:54, yaussy wrote:
>>
>>>
>>> Rob,
>>>
>>> While I don't yet have a solid / consistent test case for you, our
>>> application, using a particular test scenario the app guys have, can
>>> reproduce the problem every time.  If you have anything you want me
>>> to try
>>> out, let me know.
>>>
>>> I'm still working with my test code to get something more
>>> reproduceable,
>>> while adding some debug code to IndexManager.
>>>
>>> Kevin
>>>
>>>
>>> yaussy wrote:
>>>>
>>>> Still working on a consistent test case.  But, I've run into
>>>> something
>>>> along the way, and I'm not sure if it's related to the original
>>>> exception
>>>> I've logged here.  I was able to get the problem to happen when I
>>>> expanded
>>>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a
>>>> consumer
>>>> that consumes all those topics.  I have this configured to use a
>>>> connection for each topic, for both the publisher and the
>>>> consumer.  I've
>>>> only been able to get the original exception to happen once, but
>>>> I've more
>>>> consistently hit a problem wherein no exception is thrown, but the
>>>> broker
>>>> just stops forwarding events for one of the topics.  After this
>>>> point, the
>>>> publisher continues, but all events for this one topic are queued
>>>> to disk.
>>>>
>>>> I'm trying to see if I can reduce this test case down right now.
>>>> But,
>>>> maybe you could give this a go with the supplied consumer /
>>>> supplier test
>>>> code?
>>>>
>>>> Kevin
>>>>
>>>>
>>>> rajdavies wrote:
>>>>>
>>>>> ok - thx for checking! - let me know when you have a test case
>>>>>
>>>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>>>
>>>>>>
>>>>>> Bad news - problem still happens with "5.2" snapshot dated  
>>>>>> 4/13.  I
>>>>>> did not
>>>>>> see anymore 5.1 snapshots, but saw the 5.2 directory.  The  
>>>>>> archive
>>>>>> still has
>>>>>> 5.1 for the path names, so I figured this is what I should take.
>>>>>>
>>>>>> I'm still working with it and am trying to see if I can get any
>>>>>> more
>>>>>> debug
>>>>>> information.  I have not been able to reproduce it with a test
>>>>>> program yet.
>>>>>>
>>>>>>
>>>>>>
>>>>>> yaussy wrote:
>>>>>>>
>>>>>>> Rob,
>>>>>>>
>>>>>>> This is good news - I will try using last nights snapshot.
>>>>>>>
>>>>>>> I started out using AMQMessageStore, but I've never had good
>>>>>>> performance
>>>>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>>>>> high CPU
>>>>>>> cost as it checkpoints.  Our Durable rate requirements are  
>>>>>>> fairly
>>>>>>> significant, such as a few hundred events per second to a
>>>>>>> handful of
>>>>>>> consumers, on one of our clusters.  It has not performed well,
>>>>>>> whereas
>>>>>>> Kaha by itself is excellent.
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> rajdavies wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have not been able to reproduce this problem outside
of our
>>>>>>>>> application,
>>>>>>>>> but I'm still trying.
>>>>>>>>>
>>>>>>>>> Anyway, I'm using regular Kaha persistence (not
>>>>>>>>> AMQMessageStore),
>>>>>>>>> and during
>>>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>>>> exception
>>>>>>>>> in the
>>>>>>>>> AMQBroker, after which the broker stops forwarding events
to  
>>>>>>>>> the
>>>>>>>>> consumer.
>>>>>>>>>
>>>>>>>>> Anyone seen this?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>
>>>>>>>>> Failed
>>>>>>>>> to fill
>>>>>>>>> batch
>>>>>>>>> Stack Trace follows:
>>>>>>>>> java.lang.RuntimeException: Failed to get next index
from
>>>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>>>> 20080006, 47),
>>>>>>>>> value=(2, 20080058, 165), previousItem=7681212,  
>>>>>>>>> nextItem=7682895
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl
>>>>>>>>> .index
>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>> 267)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl 
>>>>>>>>> .container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>>>> 449)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .store
>>>>>>>>> .kahadaptor
>>>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>>>> 95)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .store
>>>>>>>>> .kahadaptor
>>>>>>>>> .KahaTopicMessageStore
>>>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .store
>>>>>>>>> .ProxyTopicMessageStore
>>>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors 
>>>>>>>>> .TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>>>> 107)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors 
>>>>>>>>> .AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>>>> 188)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>>>> 104)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors
>>>>>>>>> .StoreDurableSubscriberCursor
>>>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>>>> )
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .PrefetchSubscription 
>>>>>>>>> .dispatchPending(PrefetchSubscription.java:
>>>>>>>>> 479)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>>>> 357)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:
>>>>>>>>> 349)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>>>> 474)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:
>>>>>>>>> 194)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:

>>>>>>>>> 73)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:

>>>>>>>>> 73)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>>>> 84)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .TransportConnection 
>>>>>>>>> .processMessageAck(TransportConnection.java:
>>>>>>>>> 444)
>>>>>>>>>     at
>>>>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:
>>>>>>>>> 196)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.TransportConnection.service(TransportConnection.java:
>>>>>>>>> 293)
>>>>>>>>>     at
>>>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport
>>>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>>>> 143)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:
>>>>>>>>> 206)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>>>> 84)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>>>> 196)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>>>> 183)
>>>>>>>>>     at java.lang.Thread.run(Thread.java:619)
>>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>>     at
>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>> 383)
>>>>>>>>>     at
>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>> 361)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>>>> 46)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>>>> 66)
>>>>>>>>>     at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl
>>>>>>>>> .index
>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>> 265)
>>>>>>>>>     ... 27 more
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> View this message in context:
>>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>>>> Nabble.com.
>>>>>>>>>
>>>>>>>>
>>>>>>>> There was a bug fixed in this area last Friday - would be
>>>>>>>> interested
>>>>>>>> if this is still a problem for you with a later version?
>>>>>>>> If it is - Ill digg a little deeper.
>>>>>>>>
>>>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> -- 
>>> View this message in context:
>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16725769.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


Mime
View raw message