Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of dane@optimalsocial.com
 designates 209.85.223.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <313072CD-68FF-4AE9-B107-42D33ACD1B40@thelastpickle.com>
References: <1363203547.24795.GenericBBA@web160905.mail.bf1.yahoo.com>
	<CAL7B-cA3bwdMVRqk9XEKOzJeNiPQpR3tTnSDp3jPcrt4=JwMBQ@mail.gmail.com>
	<313072CD-68FF-4AE9-B107-42D33ACD1B40@thelastpickle.com>
Date: Thu, 14 Mar 2013 18:20:14 -0700
Message-ID: 
 <CAL7B-cAPVuBysWQ1jnBzymZaAmUmJxu51UTxWec5b9F1J21qww@mail.gmail.com>
Subject: Re: repair hangs
From: Dane Miller <dane@optimalsocial.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Mar 14, 2013 at 6:34 AM, aaron morton <aaron@thelastpickle.com> wrote:
>> 1. is this a nodetool bug?  is there any way to propagate the
>> java.io.IOException back to nodetool?
> The repair continues to work even if nodetool fails, it's a server side thing.
>
>> 2. network problems on EC2, I'm shocked!  are there recommended
>> network settings for EC2?
> Streaming does not put a timeout on the socket, in this case check the 10.82.233.59
> node to see why the pipe broke.

Good catch.  There are indeed errors on node 10.82.233.59:

cassandra3:/var/log/cassandra# grep -A1 ERROR system.log
ERROR [Thread-5571] 2013-03-11 23:38:12,554 CassandraDaemon.java (line
132) Exception in thread Thread[Thread-5571,5,main]
java.lang.UnsupportedOperationException: SSTable
/raid0/cassandra/data/OpsCenter/events_timeline/OpsCenter-events_timeline-hf-1-Data.db
is not compatible with current version ib
--
ERROR [Thread-5572] 2013-03-11 23:38:12,692 CassandraDaemon.java (line
132) Exception in thread Thread[Thread-5572,5,main]
java.lang.UnsupportedOperationException: SSTable
/raid0/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-hf-766-Data.db
is not compatible with current version ib
--
ERROR [Thread-5612] 2013-03-11 23:39:55,931 CassandraDaemon.java (line
132) Exception in thread Thread[Thread-5612,5,main]
java.lang.UnsupportedOperationException: SSTable
/raid0/cassandra/data/OpsCenter/rollups300/OpsCenter-rollups300-hf-506-Data.db
is not compatible with current version ib
--


This attempt to create a new cluster from backups of the original
cluster has not been going well.  These errors lead me to believe I
need to focus on repairing the original cluster first.

Dane