Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rev.chip@gmail.com designates
 209.85.213.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=ld3SrUoAwq/8lUudN/YODzZTJ9QU9xx4/YitLIuYFkwozDB8i0dfLjdSxYvq9pCEOC
         AplNMuJYLQDNlQk1j+WaRsmXIJXJYnAIxNqFEPn3RBWNmNxdOJcMCTElCPJeBtK6B9Wh
         fc3TJLM/p29unZiHhwaq1O7+3ON6SjSkBUbB4=
Message-ID: <4CDC9C61.6030300@gmail.com>
Date: Thu, 11 Nov 2010 17:46:09 -0800
From: Reverend Chip <rev.chip@gmail.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US;
 rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Cluster fragility
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I've been running tests with a first four-node, then eight-node
cluster.  I started with 0.7.0 beta3, but have since updated to a more
recent Hudson build.  I've been happy with a lot of things, but I've had
some really surprisingly unpleasant experiences with operational fragility.

For example, when adding four nodes to a four-node cluster (at 2x
replication), I had two nodes that insisted they were streaming data,
but no progress was made in the stream for over a day (this was with
beta3).  I had to reboot the cluster to clear that condition.  For the
purpose of making progress on other tests I decided just to reload the
data at eight-wide (with the more recent build), but if I had data I
couldn't reload or the cluster were serving in production, that would
have been a very inconvenient failure.

I also had a node that refused to bootstrap immediately, but after I
waited a day, it finally got its act together. 

I write this, not to complain per se, but to ask whether these failures
are known & expected, and rebooting a cluster is just a Thing You Have
To Do once in a while; or if not, what techniques can be used to clear
such cluster topology and streaming/replication problems without rebooting.