activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lionel Cons (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-4943) Corrupted KahaDB store: java.lang.NegativeArraySizeException
Date Thu, 19 Dec 2013 13:00:08 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852877#comment-13852877
] 

Lionel Cons commented on AMQ-4943:
----------------------------------

The previous errors are:

2013-12-18 21:12:00,482 [ActiveMQ Journal Checkpoint Worker] ERROR MessageDatabase - KahaDB
failed to store to Journal
java.io.FileNotFoundException: /activemq/data/kahadb/db-43153.log (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
        at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
        at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
        at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
2013-12-18 21:12:25,510 [ActiveMQ Journal Checkpoint Worker] INFO DefaultIOExceptionHandler
- Stopping the broker due to exception, java.io.FileNotFoundException: /activemq/data/kahadb/db-43153.log
(Too many open files)
java.io.FileNotFoundException: /activemq/data/kahadb/db-43153.log (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
        at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
        at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
        at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)

FWIW, we had "Too many open files" errors because we had too many sockets in CLOSE_WAIT state.

> Corrupted KahaDB store: java.lang.NegativeArraySizeException
> ------------------------------------------------------------
>
>                 Key: AMQ-4943
>                 URL: https://issues.apache.org/jira/browse/AMQ-4943
>             Project: ActiveMQ
>          Issue Type: Bug
>         Environment: activemq-5.5.1-fuse-10-16
>            Reporter: Lionel Cons
>            Priority: Critical
>
> One of our brokers went crazy and logged _many_ exceptions looking like this:
> 2013-12-19 12:27:09,007 [BrokerService[foobar] Task-13] ERROR AbstractStoreCursor - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@654e3615:Consumer.prod.whatever,batchResetNeeded=false,storeHasMessages=true,size=2461725,cacheEnabled=false,maxBatchSize:200
- Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Invalid location: 43153:28, : java.lang.NegativeArraySizeException
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
> 	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> 	at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1746)
> 	at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1962)
> 	at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1470)
> 	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> 	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Invalid location: 43153:28, : java.lang.NegativeArraySizeException
> 	at org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)
> 	at org.apache.kahadb.journal.Journal.read(Journal.java:601)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:908)
> 	at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1024)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:552)
> 	at org.apache.kahadb.page.Transaction.execute(Transaction.java:771)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:541)
> 	at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> 	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
> 	... 10 more
> 2013-12-19 12:27:09,007 [BrokerService[foobar] Task-13] ERROR Queue - Failed to page
in more queue messages 
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Invalid
location: 43153:28, : java.lang.NegativeArraySizeException
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
> 	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> 	at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1746)
> 	at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1962)
> 	at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1470)
> 	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> 	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: java.io.IOException: Invalid location: 43153:28,
: java.lang.NegativeArraySizeException
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
> 	... 9 more
> Caused by: java.io.IOException: Invalid location: 43153:28, : java.lang.NegativeArraySizeException
> 	at org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)
> 	at org.apache.kahadb.journal.Journal.read(Journal.java:601)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:908)
> 	at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1024)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:552)
> 	at org.apache.kahadb.page.Transaction.execute(Transaction.java:771)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:541)
> 	at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> 	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
> A restart did not hep, the KahaDB stores is corrupted :-(



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message