cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan <cne...@yahoo.com>
Subject Re: FileNotFoundException
Date Thu, 19 Mar 2015 21:01:59 GMT
HI Batranut;
In both errors you described above the files seem to be missing while compaction is running. Without
knowing what else is going on your system,  I would presume that this error occurs on this
single node only and not your entire cluster. 
Some guesses:a)  You may have a disk corruption problem.     Take the node offline and
run a diskcheck.b)   Take the node offline, wipe it clean of everything and have it rejoin
the cluster.   Check if the problem recurs.
Hope this helpsJan

 


     On Tuesday, February 24, 2015 2:56 AM, Batranut Bogdan <batranub@yahoo.com> wrote:
   

 Also I must add that grepping the logs for a particular file I see this:
 INFO [CompactionExecutor:19] 2015-02-24 10:44:35,618 CompactionTask.java (line 120) Compacting
[SSTableReader(path='/data/ranks/positions/ranks-positions-jb-339-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-354-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-408-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-286-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-20-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-127-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-357-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-257-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-316-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-41-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-285-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-338-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-180-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-398-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-249-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-284-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-294-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-248-Data.db'),
SSTableReader(path='/data/ranks/positions/ranks-positions-jb-377-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-395-Data.d
...

also several entries like this in the log after grep.java.lang.RuntimeException: java.io.FileNotFoundException:
/data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory)Caused by:
java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such
file or directory)


I was grepping for jb-41-Data.db ...  seems that this file does not exist for some reason.
I must say that when I first added the node I included it's IP in the seeds list. Then I have
decommissioned it, removed it's IP from the seed list, deleted all data / commit log / saved
caches and started it. Since then I have not manualy deleted any files . 
Any ideeas?

      On Tuesday, February 24, 2015 11:46 AM, Batranut Bogdan <batranub@yahoo.com> wrote:
   

 Hello all,
One of my C* throws a big amount of exceptions like this:

ERROR [ReadStage:792] 2015-02-24 10:43:54,183 CassandraDaemon.java (line 199) Exception in
thread Thread[ReadStage:792,5,main]java.lang.RuntimeException: java.lang.RuntimeException:
java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such
file or directory)        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008) 
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
      at java.lang.Thread.run(Thread.java:744)Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
/data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)       
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47) 
      at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48) 
      at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39) 
      at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1239) 
      at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:417) 
      at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387) 
      at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145) 
      at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45) 
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) 
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) 
      at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) 
      at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)   
    at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)     
  at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144) 
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:87) 
      at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)       
at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)     
  at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) 
      at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) 
      at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) 
      at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) 
      at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1550) 
      at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379) 
      at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) 
      at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1439) 
      at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2004) 
      ... 3 moreCaused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db
(No such file or directory)        at java.io.RandomAccessFile.open(Native Method)   
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58) 
      at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76) 
      at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43) 
      ... 29 more

I must say, that this node was started with cleared data / commit log and it was joining an
existing cluster. I see a lot of these exceptions. Not sure what to do.


    

  
Mime
View raw message