incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wright <kwri...@nanigans.com>
Subject Unable to bootstrap node
Date Tue, 06 Aug 2013 00:39:14 GMT
Hi all,

   I have been trying to bootstrap a new node into my 7 node 1.2.4 C* cluster with Vnodes
RF3 with no luck.  It gets close to completing and then the streaming just stalls with  streaming
at 99% from 1 or 2 nodes.  Nodetool netstats shows the items that have yet to stream but the
logs on the new node do not show any errors.  I tried shutting down then node, clearing all
data/commit logs/caches, and re-boot strapping with no luck.  The nodes that are hanging sending
the data only have the error below but that's related to compactions (see below) although
it is one of the files that is waiting to be sent.  I tried nodetool scrub on the column family
with the missing item but got an error indicating it could not get a hard link.  Any ideas?
 We were able to bootstrap one of the new nodes with no issues but this other one has been
a real pain.  Note that when the new node is joining the cluster, it does not appear in nodetool
status.  Is that expected?

Thanks all, my next step is to try getting a new IP for this machine, my thought being that
the cluster doesn't like me continuing to attempt to bootstrap the node repeatedly each time
getting a new host id.

[kwright@lxpcas008 ~]$ nodetool netstats | grep rts-40301_feedProducts-ib-1-Data.db
   rts: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
sections=73 progress=0/1884669 - 0%

ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java (line 174) Exception in
thread Thread[ReadStage:427,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
(No such file or directory)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
        at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
        at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
        at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
        at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
        at org.apache.cassandra.db.Table.getRow(Table.java:347)
        at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
(No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
        ... 20 more

Mime
View raw message