Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <16725769.post@talk.nabble.com>
Date: Wed, 16 Apr 2008 10:39:05 -0700 (PDT)
From: yaussy <yaussy@cboe.com>
To: users@activemq.apache.org
Subject: Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
In-Reply-To: <ECE86CB2-0AAB-4AF7-AC48-A2F1FE38B15F@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <16677512.post@talk.nabble.com>
 <D2F95C00-4469-44A1-8FBD-64F0A8DE6DA5@gmail.com>
 <16700153.post@talk.nabble.com> <16700798.post@talk.nabble.com>
 <8FAB8673-5118-4B23-AFA6-B3F10AB94666@gmail.com>
 <16711062.post@talk.nabble.com> <16721669.post@talk.nabble.com>
 <ECE86CB2-0AAB-4AF7-AC48-A2F1FE38B15F@gmail.com>


Rob,

So, I would find this under the 5.2 SNAPSHOT tomorrow morning (USA central
standard time)?

Kevin


rajdavies wrote:
> 
> yep - being able to reproduce - this is now fixed in trunk - but too  
> late for the 5.1 release
> 
> thanks,
> 
> Rob
> 
> On 16 Apr 2008, at 13:54, yaussy wrote:
> 
>>
>> Rob,
>>
>> While I don't yet have a solid / consistent test case for you, our
>> application, using a particular test scenario the app guys have, can
>> reproduce the problem every time.  If you have anything you want me  
>> to try
>> out, let me know.
>>
>> I'm still working with my test code to get something more  
>> reproduceable,
>> while adding some debug code to IndexManager.
>>
>> Kevin
>>
>>
>> yaussy wrote:
>>>
>>> Still working on a consistent test case.  But, I've run into  
>>> something
>>> along the way, and I'm not sure if it's related to the original  
>>> exception
>>> I've logged here.  I was able to get the problem to happen when I  
>>> expanded
>>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a  
>>> consumer
>>> that consumes all those topics.  I have this configured to use a
>>> connection for each topic, for both the publisher and the  
>>> consumer.  I've
>>> only been able to get the original exception to happen once, but  
>>> I've more
>>> consistently hit a problem wherein no exception is thrown, but the  
>>> broker
>>> just stops forwarding events for one of the topics.  After this  
>>> point, the
>>> publisher continues, but all events for this one topic are queued  
>>> to disk.
>>>
>>> I'm trying to see if I can reduce this test case down right now.   
>>> But,
>>> maybe you could give this a go with the supplied consumer /  
>>> supplier test
>>> code?
>>>
>>> Kevin
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>> ok - thx for checking! - let me know when you have a test case
>>>>
>>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>>
>>>>>
>>>>> Bad news - problem still happens with "5.2" snapshot dated 4/13.  I
>>>>> did not
>>>>> see anymore 5.1 snapshots, but saw the 5.2 directory.  The archive
>>>>> still has
>>>>> 5.1 for the path names, so I figured this is what I should take.
>>>>>
>>>>> I'm still working with it and am trying to see if I can get any  
>>>>> more
>>>>> debug
>>>>> information.  I have not been able to reproduce it with a test
>>>>> program yet.
>>>>>
>>>>>
>>>>>
>>>>> yaussy wrote:
>>>>>>
>>>>>> Rob,
>>>>>>
>>>>>> This is good news - I will try using last nights snapshot.
>>>>>>
>>>>>> I started out using AMQMessageStore, but I've never had good
>>>>>> performance
>>>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>>>> high CPU
>>>>>> cost as it checkpoints.  Our Durable rate requirements are fairly
>>>>>> significant, such as a few hundred events per second to a  
>>>>>> handful of
>>>>>> consumers, on one of our clusters.  It has not performed well,
>>>>>> whereas
>>>>>> Kaha by itself is excellent.
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> rajdavies wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> I have not been able to reproduce this problem outside of our
>>>>>>>> application,
>>>>>>>> but I'm still trying.
>>>>>>>>
>>>>>>>> Anyway, I'm using regular Kaha persistence (not  
>>>>>>>> AMQMessageStore),
>>>>>>>> and during
>>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>>> exception
>>>>>>>> in the
>>>>>>>> AMQBroker, after which the broker stops forwarding events to the
>>>>>>>> consumer.
>>>>>>>>
>>>>>>>> Anyone seen this?
>>>>>>>>
>>>>>>>>
>>>>>>>> <
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>  
>>>>>>>> Failed
>>>>>>>> to fill
>>>>>>>> batch
>>>>>>>> Stack Trace follows:
>>>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>>> 20080006, 47),
>>>>>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl
>>>>>>>> .index 
>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>> 267)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>>> 449)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .store
>>>>>>>> .kahadaptor 
>>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>>> 95)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .store
>>>>>>>> .kahadaptor
>>>>>>>> .KahaTopicMessageStore
>>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .store
>>>>>>>> .ProxyTopicMessageStore
>>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>>> 107)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>>> 188)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>>> 104)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors
>>>>>>>> .StoreDurableSubscriberCursor
>>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>>> )
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
>>>>>>>> 479)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region 
>>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>>> 357)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java: 
>>>>>>>> 349)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq 
>>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>>> 474)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java: 
>>>>>>>> 194)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker 
>>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>>> 84)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .TransportConnection.processMessageAck(TransportConnection.java:
>>>>>>>> 444)
>>>>>>>>      at
>>>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java: 
>>>>>>>> 196)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.TransportConnection.service(TransportConnection.java: 
>>>>>>>> 293)
>>>>>>>>      at
>>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport
>>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>>> 143)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java: 
>>>>>>>> 206)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>>> 84)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>>> 196)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache 
>>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>>> 183)
>>>>>>>>      at java.lang.Thread.run(Thread.java:619)
>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>      at  
>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>> 383)
>>>>>>>>      at  
>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>> 361)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha 
>>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>>> 46)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq 
>>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>>> 66)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl
>>>>>>>> .index 
>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>> 265)
>>>>>>>>      ... 27 more
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>>> Sent from the ActiveMQ - User mailing list archive at  
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>
>>>>>>> There was a bug fixed in this area last Friday - would be  
>>>>>>> interested
>>>>>>> if this is still a problem for you with a later version?
>>>>>>> If it is - Ill digg a little deeper.
>>>>>>>
>>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> -- 
>>>>> View this message in context:
>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16725769.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.