activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Black (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-3503) Too many open files for db log
Date Fri, 16 Sep 2011 14:32:10 GMT

    [ https://issues.apache.org/jira/browse/AMQ-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106092#comment-13106092
] 

Michael Black commented on AMQ-3503:
------------------------------------

The db log files seem to be getting deleted now...whereas before (in 5.5.0) all the db-*-.log
files were still in the directory (over 1000 of them).

So...for right now the latest snapshot seems to have fixed this.

I'll follow up later after a few billion messages have been put in the queue and it's still
running.

That will be in a day or two.

Total files open by apachemq seems stable at around 212-215

lsof | grep apache | wc -l



ls -l
total 96360
-rw-rw-r-- 1 root root 33554649 Sep 16 10:21 db-44.log
-rw-rw-r-- 1 root root 33554465 Sep 16 10:21 db-45.log
-rw-rw-r-- 1 root root 33030144 Sep 16 10:21 db-46.log
-rw-rw-r-- 1 root root        0 Sep 16 10:17 lock
-rw-rw-r-- 1 root root    12288 Sep 16 10:17 tmpDB.data
-rw-rw-r-- 1 root root    12304 Sep 16 10:17 tmpDB.redo

Michael D. Black

Senior Scientist

NG Information Systems

Advanced Analytics Directorate





> Too many open files for db log
> ------------------------------
>
>                 Key: AMQ-3503
>                 URL: https://issues.apache.org/jira/browse/AMQ-3503
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.0
>         Environment: Redhat 5.7.  Default conf file used for startup
>            Reporter: Michael Black
>            Priority: Blocker
>
> lsof | grep apache | grep data/localhost | wc -l
> Shows constantly increasing number of open files. Had 1016 open when it died.
> ulimit is set at 1024.
> Docs say files are supposed to be removed when "no longer needed".
> We're running 3 producers and 3 consumers so no messages should be left in the queue.
> At the point of dying we have put in around 2 billion messages.
> 2011-09-16 00:18:21,148 | ERROR | I/O error | org.apache.activemq.broker.region.cursors.FilePendingMessageCursor
| Queue:MCNA
> java.io.FileNotFoundException: /usr/local/apache-activemq-5.5.0/data/localhost/tmp_storage/db-1987.log
(Too many open files)
>   at java.io.RandomAccessFile.open(Native Method)
>   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>   at org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
>   at org.apache.kahadb.journal.DataFileAccessor.<init>(DataFileAccessor.java:49)
>   at org.apache.kahadb.journal.DataFileAccessorPool$Pool.openDataFileReader(DataFileAccessorPool.java:53)
>   at org.apache.kahadb.journal.DataFileAccessorPool.openDataFileAccessor(DataFileAccessorPool.java:139)
>   at org.apache.kahadb.journal.Journal.read(Journal.java:598)
>   at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
>   at org.apache.activemq.store.kahadb.plist.PList.getNext(PList.java:316)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:500)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.next(FilePendingMessageCursor.java:473)
>   at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.next(FilePendingMessageCursor.java:293)
>   at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:135)
>   at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1714)
>   at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
>   at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
>   at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
>   at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message