incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sankalp kohli <kohlisank...@gmail.com>
Subject Re: Unable to bootstrap node
Date Tue, 06 Aug 2013 19:03:31 GMT
@Aaron
This problem happens when you drop and recreate a keyspace with the same
name and you do it very quickly. I have also filed a JIRA for it

https://issues.apache.org/jira/browse/CASSANDRA-5843


On Tue, Aug 6, 2013 at 10:31 AM, Keith Wright <kwright@nanigans.com> wrote:

> The file does not appear on disk and the permissions are definitely
> correct.  We have seen the file in snapshots.   This is completely blocking
> us from adding the new node.  How can we recover?  Just run repairs?
>
> Thanks
>
> From: Aaron Morton <aaron@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Date: Tuesday, August 6, 2013 4:06 AM
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Subject: Re: Unable to bootstrap node
>
> Caused by: java.io.FileNotFoundException:
> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
> (No such file or directory)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>         at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
>         at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>
>
> This is somewhat serous, specially if it's from the a bug in dropping
> tables. Though I would expect that would show up for a lot of people.
>
> Does the file exist on disk?
> Are the permissions correct ?
>
> IMHO you need to address this issue on the existing nodes before worrying
> about the new node.
>
> Cheers
>
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6/08/2013, at 1:25 PM, sankalp kohli <kohlisankalp@gmail.com> wrote:
>
> Let me know if this fixes the problem?
>
>
> On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisankalp@gmail.com>wrote:
>
>> So the problem is that when you dropped and recreated the table with the
>> same name, some how the old CFStore object was not purged. So now there
>> were two objects which caused same sstable to have 2 SSTableReader object.
>>
>> The fix is to find all nodes which is emitting this FileNotFound
>> Exception and restart them.
>>
>> In your case, restart the node which is serving the data and emitting
>> FileNotFound exception.
>>
>> Once this is up, again restart the bootstrapping node with bootstrap
>> argument. Now it will successfully stream the data.
>>
>>
>> On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwright@nanigans.com>wrote:
>>
>>>
>>> Yes we likely dropped and recreated tables.  If we stop the sending node, what
will happen to the bootstrapping node?
>>>
>>> sankalp kohli <kohlisankalp@gmail.com> wrote:
>>>
>>>
>>> Hi,
>>>     The problem is that the node sending the stream is hitting this
>>> FileNotFound exception. You need to restart this node and it should fix the
>>> problem.
>>>
>>> Are you seeing lot of FileNotFoundExceptions? Did you do any schema
>>> change recently?
>>>
>>> Sankalp
>>>
>>>
>>> On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwright@nanigans.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>>    I have been trying to bootstrap a new node into my 7 node 1.2.4 C*
>>>> cluster with Vnodes RF3 with no luck.  It gets close to completing and then
>>>> the streaming just stalls with  streaming at 99% from 1 or 2 nodes.
>>>>  Nodetool netstats shows the items that have yet to stream but the logs on
>>>> the new node do not show any errors.  I tried shutting down then node,
>>>> clearing all data/commit logs/caches, and re-boot strapping with no luck.
>>>>  The nodes that are hanging sending the data only have the error below but
>>>> that's related to compactions (see below) although it is one of the files
>>>> that is waiting to be sent.  I tried nodetool scrub on the column family
>>>> with the missing item but got an error indicating it could not get a hard
>>>> link.  Any ideas?  We were able to bootstrap one of the new nodes with no
>>>> issues but this other one has been a real pain.  Note that when the new
>>>> node is joining the cluster, it does not appear in nodetool status.  Is
>>>> that expected?
>>>>
>>>> Thanks all, my next step is to try getting a new IP for this machine,
>>>> my thought being that the cluster doesn't like me continuing to attempt to
>>>> bootstrap the node repeatedly each time getting a new host id.
>>>>
>>>> [kwright@lxpcas008 ~]$ nodetool netstats | grep
>>>> rts-40301_feedProducts-ib-1-Data.db
>>>>    rts:
>>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
>>>> sections=73 progress=0/1884669 - 0%
>>>>
>>>> ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java
>>>> (line 174) Exception in thread Thread[ReadStage:427,5,main]
>>>> java.lang.RuntimeException: java.io.FileNotFoundException:
>>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
>>>> (No such file or directory)
>>>>         at
>>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
>>>>         at
>>>> org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
>>>>         at
>>>> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
>>>>         at
>>>> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
>>>>         at
>>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
>>>>         at
>>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
>>>>         at
>>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
>>>>         at
>>>> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
>>>>         at
>>>> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>>>>         at
>>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
>>>>         at
>>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>>>>         at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
>>>>         at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>>>>         at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
>>>>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>>>>         at
>>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
>>>>         at
>>>> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
>>>>         at
>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>         at java.lang.Thread.run(Thread.java:722)
>>>> Caused by: java.io.FileNotFoundException:
>>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db
>>>> (No such file or directory)
>>>>         at java.io.RandomAccessFile.open(Native Method)
>>>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>>>>         at
>>>> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
>>>>         at
>>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>>>>         at
>>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
>>>>         ... 20 more
>>>>
>>>
>>>
>>
>
>

Mime
View raw message