incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franc Carter <franc.car...@sirca.org.au>
Subject Re: stalled bootstrap
Date Wed, 15 Feb 2012 10:58:07 GMT
On Wed, Feb 15, 2012 at 10:21 AM, aaron morton <aaron@thelastpickle.com>wrote:

> The assertion looks like a bug.
>
> Can you run it with DEBUG logging ?
>

Sorry - I had to blow the instances away. I'm on a tight timeline for the
Proof of Concept I am doing and rebuilding a 4-node cluster from scratch
was going to be way faster. If I get time I'll try to reproduce it towards
the end of the project - sorry.



> Do you have compression enabled ?
>

Yes - SnappyCompressor


>
> Can you please submit a ticket here
> https://issues.apache.org/jira/browse/CASSANDRA with the extra info and
> update the email thread.
>

Would you still like this even though I can't get much detail ?


>
> I *think* that the node this is happening on is failing to create the temp
> file in IncomingStreamReader.streamIn and the it's then trying to delete
> the file before it retries. Extra debugging may be a help. The assertion is
> hiding the original error.
>
> Can you check if the new node code create files in the data directory ?
>
>
I'll try these if I can get time to retest - thanks for the pointers

cheers


> Cheers
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/02/2012, at 12:42 AM, Franc Carter wrote:
>
>
> Hi,
>
> I'm running the DataSatx 1.0.7 AMI in ec2. I started with two nodes and
> have just added a third node on the way to expanding to a four node cluster.
>
> The bootstrapping was going along ok for a while, but has stalled. In
> /var/log/cassandra/system.log I am seeing this repeated continuously (tmp
> file changes each time)
>
> INFO [Thread-529373] 2012-02-14 11:36:18,350 StreamInSession.java (line
> 120) Streaming of file
> /raid0/cassandra/data/OpsCenter/rollups7200-hc-1-Data.db sections=2
> progress=0/42387 - 0% from
> org.apache.cassandra.streaming.StreamInSession@6ebcf58a failed:
> requesting a retry.
> ERROR [Thread-529373] 2012-02-14 11:36:18,351 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-529373,5,main]
> java.lang.AssertionError: attempted to delete non-existing file
> rollups7200-tmp-hc-529319-Data.db
>         at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:172)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:92)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
>
>
> Any advice on how to resolve this ?
>
> thanks
>
> --
> *Franc Carter* | Systems architect | Sirca Ltd
>  <marc.zianideferranti@sirca.org.au>
> franc.carter@sirca.org.au | www.sirca.org.au
> Tel: +61 2 9236 9118
>  Level 9, 80 Clarence St, Sydney NSW 2000
> PO Box H58, Australia Square, Sydney NSW 1215
>
>
>


-- 

*Franc Carter* | Systems architect | Sirca Ltd
 <marc.zianideferranti@sirca.org.au>

franc.carter@sirca.org.au | www.sirca.org.au

Tel: +61 2 9236 9118

Level 9, 80 Clarence St, Sydney NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215

Mime
View raw message