incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: mutation stalls and FileNotFoundException
Date Thu, 09 May 2013 22:35:40 GMT
When mutation stage messages are dropped I start by looking for GC problems in the cassandra
logs.

You also have some blocked flush writer tasks, which may be due to a large number of CF's,
a large number of secondary indexes, slow disk IO or excessive use of snapshot / flush. Check
the comments for the memtable_flush_queue_size in the yaml file. 

Hope that helps. 
 
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 8/05/2013, at 4:41 AM, Keith Wright <kwright@nanigans.com> wrote:

> I am running 1.2.4 with Vnodes and have been writing at low volume.  I have doubled the
volume and suddenly 3 of my 6 nodes are showing much higher load than the others (30 vs 3)
and tpstats show the mutation stage as completely full (see below).  I did find a FileNotFoundException
that I pasted below which appears to be caused by creating, dropping, and creating a keyspace
(something I did but 4 or 5 days ago).  Anyone have any idea what's going on here?
> 
> Thanks
> 
> Keiths-MacBook-Pro:bin keith$ ./nodetool tpstats -h lxpcas005.nanigans.com
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         0         0         130990         0              
  0
> RequestResponseStage              0         0         344216         0              
  0
> MutationStage                   128       452        3464036         0              
  0
> ReadRepairStage                   0         0          14131         0              
  0
> ReplicateOnWriteStage             0         0          32872         0              
  0
> GossipStage                       1       611           6351         0              
  0
> AntiEntropyStage                  0         0              0         0              
  0
> MigrationStage                    0         0              9         0              
  0
> MemtablePostFlusher               0         0             91         0              
  0
> FlushWriter                       0         0             60         0              
 27
> MiscStage                         0         0              0         0              
  0
> commitlog_archiver                0         0              0         0              
  0
> InternalResponseStage             0         0              3         0              
  0
> HintedHandoff                     1         1             13         0              
  0
> 
> Message type           Dropped
> RANGE_SLICE                  0
> READ_REPAIR                 54
> BINARY                       0
> READ                         0
> MUTATION                  8539
> _TRACE                       0
> REQUEST_RESPONSE             0
> 
> 
> 
> ERROR [ReplicateOnWriteStage:95404] 2013-05-06 14:55:06,555 CassandraDaemon.java (line
174) Exception in thread Thread[ReplicateOnWriteStage:95404,5,main]
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException:
/data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No
such file or directory)
>         at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
>         at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
>         at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
>         at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
>         at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
>         at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
>         at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
>         at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
>         at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>         at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
>         at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
>         at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
>         at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
>         ... 3 more
> Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
>         ... 21 more


Mime
View raw message