incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Unable to bootstrap node
Date Wed, 07 Aug 2013 17:57:59 GMT
Thanks for the update :)

A

-----------------
Aaron Morton
Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 7/08/2013, at 7:03 AM, sankalp kohli <kohlisankalp@gmail.com> wrote:

> @Aaron
> This problem happens when you drop and recreate a keyspace with the same name and you
do it very quickly. I have also filed a JIRA for it
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5843
> 
> 
> On Tue, Aug 6, 2013 at 10:31 AM, Keith Wright <kwright@nanigans.com> wrote:
> The file does not appear on disk and the permissions are definitely correct.  We have
seen the file in snapshots.   This is completely blocking us from adding the new node.  How
can we recover?  Just run repairs?
> 
> Thanks
> 
> From: Aaron Morton <aaron@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Date: Tuesday, August 6, 2013 4:06 AM
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Subject: Re: Unable to bootstrap node
> 
>> Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
(No such file or directory)
>>         at java.io.RandomAccessFile.open(Native Method)
>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
>>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>
> This is somewhat serous, specially if it's from the a bug in dropping tables. Though
I would expect that would show up for a lot of people. 
> 
> Does the file exist on disk?
> Are the permissions correct ? 
> 
> IMHO you need to address this issue on the existing nodes before worrying about the new
node. 
> 
> Cheers
>  
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 6/08/2013, at 1:25 PM, sankalp kohli <kohlisankalp@gmail.com> wrote:
> 
>> Let me know if this fixes the problem?
>> 
>> 
>> On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisankalp@gmail.com> wrote:
>> So the problem is that when you dropped and recreated the table with the same name,
some how the old CFStore object was not purged. So now there were two objects which caused
same sstable to have 2 SSTableReader object. 
>> 
>> The fix is to find all nodes which is emitting this FileNotFound Exception and restart
them. 
>> 
>> In your case, restart the node which is serving the data and emitting FileNotFound
exception. 
>> 
>> Once this is up, again restart the bootstrapping node with bootstrap argument. Now
it will successfully stream the data. 
>> 
>> 
>> On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwright@nanigans.com> wrote:
>> 
>> Yes we likely dropped and recreated tables.  If we stop the sending node, what will
happen to the bootstrapping node?
>> 
>> sankalp kohli <kohlisankalp@gmail.com> wrote:
>> 
>> Hi,
>>     The problem is that the node sending the stream is hitting this FileNotFound
exception. You need to restart this node and it should fix the problem. 
>> 
>> Are you seeing lot of FileNotFoundExceptions? Did you do any schema change recently?
>> 
>> Sankalp
>> 
>> 
>> On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwright@nanigans.com> wrote:
>> Hi all,
>> 
>>    I have been trying to bootstrap a new node into my 7 node 1.2.4 C* cluster with
Vnodes RF3 with no luck.  It gets close to completing and then the streaming just stalls with
 streaming at 99% from 1 or 2 nodes.  Nodetool netstats shows the items that have yet to stream
but the logs on the new node do not show any errors.  I tried shutting down then node, clearing
all data/commit logs/caches, and re-boot strapping with no luck.  The nodes that are hanging
sending the data only have the error below but that's related to compactions (see below) although
it is one of the files that is waiting to be sent.  I tried nodetool scrub on the column family
with the missing item but got an error indicating it could not get a hard link.  Any ideas?
 We were able to bootstrap one of the new nodes with no issues but this other one has been
a real pain.  Note that when the new node is joining the cluster, it does not appear in nodetool
status.  Is that expected?
>> 
>> Thanks all, my next step is to try getting a new IP for this machine, my thought
being that the cluster doesn't like me continuing to attempt to bootstrap the node repeatedly
each time getting a new host id.
>> 
>> [kwright@lxpcas008 ~]$ nodetool netstats | grep rts-40301_feedProducts-ib-1-Data.db
>>    rts: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
sections=73 progress=0/1884669 - 0%
>> 
>> ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java (line 174) Exception
in thread Thread[ReadStage:427,5,main]
>> java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
(No such file or directory)
>>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
>>         at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
>>         at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
>>         at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
>>         at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
>>         at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
>>         at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
>>         at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
>>         at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>>         at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
>>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
>>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
>>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>>         at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
>>         at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
>>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
(No such file or directory)
>>         at java.io.RandomAccessFile.open(Native Method)
>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
>>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
>>         ... 20 more
>> 
>> 
>> 
> 
> 


Mime
View raw message