incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Freeman <8fo...@gmail.com>
Subject FileNotFoundException while inserting (1.2.8)
Date Tue, 10 Sep 2013 15:40:44 GMT
While running a heavy insert load, one of my nodes started throwing this 
exception when trying a compaction:

  INFO [CompactionExecutor:23] 2013-09-09 16:08:07,528 
CompactionTask.java (line 105) Compacting [SSTableReader(p
ath='/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-6-Data.db'), 
SSTableReader(path='/var/lib/cassandr
a/data/smdb/tracedata/smdb-tracedata-ic-5-Data.db'), 
SSTableReader(path='/var/lib/cassandra/data/smdb/tracedata/
smdb-tracedata-ic-1-Data.db'), 
SSTableReader(path='/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-4-Da
ta.db'), 
SSTableReader(path='/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-2-Data.db')]
ERROR [CompactionExecutor:23] 2013-09-09 16:08:07,611 
CassandraDaemon.java (line 192) Exception in thread Thread
[CompactionExecutor:23,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: 
/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata
-ic-5-Data.db (No such file or directory)
         at 
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
         at 
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1194)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
         at 
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1014)
         at 
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1026)
         at 
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.
java:157)
         at 
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
         at 
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
         at 
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
         at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
         at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
         at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
         at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
         at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
         at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: 
/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-5-Data.db (No 
such file or directory)
         at java.io.RandomAccessFile.open(Native Method)
         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
         at 
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
         at 
org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
         at 
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
         ... 18 more

This shows up many times in the log.  I figured running a repair on the 
node might fix it, but the repair ran for over an hour (the node only 
has about 3G of data), so I figured it was hung.  I tried restarting the 
repair, but each time it starts the node logs that same exception 
immediately:

  INFO [AntiEntropySessions:5] 2013-09-10 09:36:35,526 
AntiEntropyService.java (line 651) [repair #c6ab9c00-1a2e-
11e3-b0e5-05d1729cecff] new session: will sync /192.168.27.73, 
/192.168.27.75 on range (4925454539472655923,4991
066214171147775] for smdb.[tracedata, processors]
  INFO [AntiEntropySessions:5] 2013-09-10 09:36:35,526 
AntiEntropyService.java (line 857) [repair #c6ab9c00-1a2e-
11e3-b0e5-05d1729cecff] requesting merkle trees for tracedata (to 
[/192.168.27.75, /192.168.27.73])
ERROR [ValidationExecutor:2] 2013-09-10 09:36:35,535 
CassandraDaemon.java (line 192) Exception in thread Thread[
ValidationExecutor:2,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: 
/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata
-ic-5-Data.db (No such file or directory)
         at 
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
         at 
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1194)
         ...

How can I fix this node?




Mime
View raw message