activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yaussy <yau...@cboe.com>
Subject Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Date Tue, 15 Apr 2008 21:29:06 GMT

Still working on a consistent test case.  But, I've run into something along
the way, and I'm not sure if it's related to the original exception I've
logged here.  I was able to get the problem to happen when I expanded my
test case to a publisher of 6 topics, at 100+/sec/topic, and a consumer that
consumes all those topics.  I have this configured to use a connection for
each topic, for both the publisher and the consumer.  I've only been able to
get the original exception to happen once, but I've more consistently hit a
problem wherein no exception is thrown, but the broker just stops forwarding
events for one of the topics.  After this point, the publisher continues,
but all events for this one topic are queued to disk.

I'm trying to see if I can reduce this test case down right now.  But, maybe
you could give this a go with the supplied consumer / supplier test code?

Kevin


rajdavies wrote:
> 
> ok - thx for checking! - let me know when you have a test case
> 
> On 15 Apr 2008, at 15:06, yaussy wrote:
> 
>>
>> Bad news - problem still happens with "5.2" snapshot dated 4/13.  I  
>> did not
>> see anymore 5.1 snapshots, but saw the 5.2 directory.  The archive  
>> still has
>> 5.1 for the path names, so I figured this is what I should take.
>>
>> I'm still working with it and am trying to see if I can get any more  
>> debug
>> information.  I have not been able to reproduce it with a test  
>> program yet.
>>
>>
>>
>> yaussy wrote:
>>>
>>> Rob,
>>>
>>> This is good news - I will try using last nights snapshot.
>>>
>>> I started out using AMQMessageStore, but I've never had good  
>>> performance
>>> luck with AMQs Journal - it always seems to have this frequent,  
>>> high CPU
>>> cost as it checkpoints.  Our Durable rate requirements are fairly
>>> significant, such as a few hundred events per second to a handful of
>>> consumers, on one of our clusters.  It has not performed well,  
>>> whereas
>>> Kaha by itself is excellent.
>>>
>>> Kevin
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>>
>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>
>>>>>
>>>>> I have not been able to reproduce this problem outside of our
>>>>> application,
>>>>> but I'm still trying.
>>>>>
>>>>> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
>>>>> and during
>>>>> a Durable Topic test, I'm eventually getting the following  
>>>>> exception
>>>>> in the
>>>>> AMQBroker, after which the broker stops forwarding events to the
>>>>> consumer.
>>>>>
>>>>> Anyone seen this?
>>>>>
>>>>>
>>>>> <
>>>>> org
>>>>> .apache 
>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
>>>>> to fill
>>>>> batch
>>>>> Stack Trace follows:
>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>> 20080006, 47),
>>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha
>>>>> .impl
>>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java: 
>>>>> 267)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha 
>>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>> 449)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .store
>>>>> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java: 
>>>>> 95)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .store
>>>>> .kahadaptor
>>>>> .KahaTopicMessageStore
>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .store
>>>>> .ProxyTopicMessageStore
>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java: 
>>>>> 107)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java: 
>>>>> 188)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region 
>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>> 104)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors
>>>>> .StoreDurableSubscriberCursor
>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>> )
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java: 
>>>>> 479)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>> 357)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>> 474)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>>>>>       at
>>>>> org
>>>>> .apache 
>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>       at
>>>>> org
>>>>> .apache 
>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java: 
>>>>> 84)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .TransportConnection.processMessageAck(TransportConnection.java: 
>>>>> 444)
>>>>>       at
>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.TransportConnection.service(TransportConnection.java:293)
>>>>>       at
>>>>> org.apache.activemq.broker.TransportConnection
>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq 
>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport 
>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>> 143)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq 
>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>> 84)
>>>>>       at
>>>>> org
>>>>> .apache 
>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>> 196)
>>>>>       at
>>>>> org 
>>>>> .apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>> 183)
>>>>>       at java.lang.Thread.run(Thread.java:619)
>>>>> Caused by: java.io.EOFException
>>>>>       at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>> 383)
>>>>>       at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>> 361)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java: 
>>>>> 46)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java: 
>>>>> 66)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha
>>>>> .impl
>>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java: 
>>>>> 265)
>>>>>       ... 27 more
>>>>>
>>>>> -- 
>>>>> View this message in context:
>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>> There was a bug fixed in this area last Friday - would be interested
>>>> if this is still a problem for you with a later version?
>>>> If it is - Ill digg a little deeper.
>>>>
>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>
>>>>
>>>>
>>>>
>>>> cheers,
>>>>
>>>> Rob
>>>>
>>>> http://open.iona.com/ -Enterprise Open Integration
>>>> http://rajdavies.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16711062.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message