incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sankalp kohli <kohlisank...@gmail.com>
Subject Re: Unable to bootstrap node
Date Tue, 06 Aug 2013 01:25:41 GMT
Let me know if this fixes the problem?


On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisankalp@gmail.com>wrote:

> So the problem is that when you dropped and recreated the table with the
> same name, some how the old CFStore object was not purged. So now there
> were two objects which caused same sstable to have 2 SSTableReader object.
>
> The fix is to find all nodes which is emitting this FileNotFound Exception
> and restart them.
>
> In your case, restart the node which is serving the data and emitting
> FileNotFound exception.
>
> Once this is up, again restart the bootstrapping node with bootstrap
> argument. Now it will successfully stream the data.
>
>
>  On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwright@nanigans.com>wrote:
>
>>  Yes we likely dropped and recreated tables.  If we stop the sending node, what will
happen to the bootstrapping node?
>>
>> sankalp kohli <kohlisankalp@gmail.com> wrote:
>>
>>
>>  Hi,
>>     The problem is that the node sending the stream is hitting this
>> FileNotFound exception. You need to restart this node and it should fix the
>> problem.
>>
>>  Are you seeing lot of FileNotFoundExceptions? Did you do any schema
>> change recently?
>>
>>  Sankalp
>>
>>
>> On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwright@nanigans.com>wrote:
>>
>>>  Hi all,
>>>
>>>     I have been trying to bootstrap a new node into my 7 node 1.2.4 C*
>>> cluster with Vnodes RF3 with no luck.  It gets close to completing and then
>>> the streaming just stalls with  streaming at 99% from 1 or 2 nodes.
>>>  Nodetool netstats shows the items that have yet to stream but the logs on
>>> the new node do not show any errors.  I tried shutting down then node,
>>> clearing all data/commit logs/caches, and re-boot strapping with no luck.
>>>  The nodes that are hanging sending the data only have the error below but
>>> that's related to compactions (see below) although it is one of the files
>>> that is waiting to be sent.  I tried nodetool scrub on the column family
>>> with the missing item but got an error indicating it could not get a hard
>>> link.  Any ideas?  We were able to bootstrap one of the new nodes with no
>>> issues but this other one has been a real pain.  Note that when the new
>>> node is joining the cluster, it does not appear in nodetool status.  Is
>>> that expected?
>>>
>>>  Thanks all, my next step is to try getting a new IP for this machine,
>>> my thought being that the cluster doesn't like me continuing to attempt to
>>> bootstrap the node repeatedly each time getting a new host id.
>>>
>>>  [kwright@lxpcas008 ~]$ nodetool netstats | grep
>>> rts-40301_feedProducts-ib-1-Data.db
>>>    rts:
>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
>>> sections=73 progress=0/1884669 - 0%
>>>
>>>  ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java
>>> (line 174) Exception in thread Thread[ReadStage:427,5,main]
>>> java.lang.RuntimeException: java.io.FileNotFoundException:
>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
>>> (No such file or directory)
>>>         at
>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
>>>         at
>>> org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
>>>         at
>>> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
>>>         at
>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
>>>         at
>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
>>>         at
>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
>>>         at
>>> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
>>>         at
>>> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>>>         at
>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
>>>         at
>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>>>         at
>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
>>>         at
>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>>>         at
>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
>>>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>>>         at
>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
>>>         at
>>> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
>>>         at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>         at java.lang.Thread.run(Thread.java:722)
>>> Caused by: java.io.FileNotFoundException:
>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
>>> (No such file or directory)
>>>         at java.io.RandomAccessFile.open(Native Method)
>>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>>>         at
>>> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
>>>         at
>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>>>         at
>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
>>>         ... 20 more
>>>
>>
>>
>

Mime
View raw message