cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: stalled bootstrap
Date Tue, 14 Feb 2012 23:21:10 GMT
The assertion looks like a bug. 

Can you run it with DEBUG logging ? 
Do you have compression enabled ? 

Can you please submit a ticket here https://issues.apache.org/jira/browse/CASSANDRA with the
extra info and update the email thread. 

I *think* that the node this is happening on is failing to create the temp file in IncomingStreamReader.streamIn
and the it's then trying to delete the file before it retries. Extra debugging may be a help.
The assertion is hiding the original error.

Can you check if the new node code create files in the data directory ? 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 15/02/2012, at 12:42 AM, Franc Carter wrote:

> 
> Hi,
> 
> I'm running the DataSatx 1.0.7 AMI in ec2. I started with two nodes and have just added
a third node on the way to expanding to a four node cluster.
> 
> The bootstrapping was going along ok for a while, but has stalled. In /var/log/cassandra/system.log
I am seeing this repeated continuously (tmp file changes each time)
> 
> INFO [Thread-529373] 2012-02-14 11:36:18,350 StreamInSession.java (line 120) Streaming
of file /raid0/cassandra/data/OpsCenter/rollups7200-hc-1-Data.db sections=2 progress=0/42387
- 0% from org.apache.cassandra.streaming.StreamInSession@6ebcf58a failed: requesting a retry.
> ERROR [Thread-529373] 2012-02-14 11:36:18,351 AbstractCassandraDaemon.java (line 139)
Fatal exception in thread Thread[Thread-529373,5,main]
> java.lang.AssertionError: attempted to delete non-existing file rollups7200-tmp-hc-529319-Data.db
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
>         at org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:172)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:92)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
> 
> 
> Any advice on how to resolve this ?
> 
> thanks
> 
> -- 
> Franc Carter | Systems architect | Sirca Ltd
> franc.carter@sirca.org.au | www.sirca.org.au
> Tel: +61 2 9236 9118 
> Level 9, 80 Clarence St, Sydney NSW 2000
> PO Box H58, Australia Square, Sydney NSW 1215
> 


Mime
View raw message