incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wright <kwri...@nanigans.com>
Subject mutation stalls and FileNotFoundException
Date Tue, 07 May 2013 16:41:48 GMT
I am running 1.2.4 with Vnodes and have been writing at low volume.  I have doubled the volume
and suddenly 3 of my 6 nodes are showing much higher load than the others (30 vs 3) and tpstats
show the mutation stage as completely full (see below).  I did find a FileNotFoundException
that I pasted below which appears to be caused by creating, dropping, and creating a keyspace
(something I did but 4 or 5 days ago).  Anyone have any idea what's going on here?

Thanks

Keiths-MacBook-Pro:bin keith$ ./nodetool tpstats -h lxpcas005.nanigans.com
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
ReadStage                         0         0         130990         0                 0
RequestResponseStage              0         0         344216         0                 0
MutationStage                   128       452        3464036         0                 0
ReadRepairStage                   0         0          14131         0                 0
ReplicateOnWriteStage             0         0          32872         0                 0
GossipStage                       1       611           6351         0                 0
AntiEntropyStage                  0         0              0         0                 0
MigrationStage                    0         0              9         0                 0
MemtablePostFlusher               0         0             91         0                 0
FlushWriter                       0         0             60         0                27
MiscStage                         0         0              0         0                 0
commitlog_archiver                0         0              0         0                 0
InternalResponseStage             0         0              3         0                 0
HintedHandoff                     1         1             13         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                 54
BINARY                       0
READ                         0
MUTATION                  8539
_TRACE                       0
REQUEST_RESPONSE             0



ERROR [ReplicateOnWriteStage:95404] 2013-05-06 14:55:06,555 CassandraDaemon.java (line 174)
Exception in thread Thread[ReplicateOnWriteStage:95404,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
        at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
        at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
        at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
        at org.apache.cassandra.db.Table.getRow(Table.java:347)
        at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
        at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
        at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
        ... 3 more
Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
        ... 21 more

Mime
View raw message