From user-return-36431-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Sep 10 01:08:59 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 99DBE1091C for ; Tue, 10 Sep 2013 01:08:59 +0000 (UTC) Received: (qmail 41174 invoked by uid 500); 10 Sep 2013 01:08:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41122 invoked by uid 500); 10 Sep 2013 01:08:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41114 invoked by uid 99); 10 Sep 2013 01:08:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Sep 2013 01:08:56 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of alex.heneveld@cloudsoftcorp.com designates 74.125.82.44 as permitted sender) Received: from [74.125.82.44] (HELO mail-wg0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Sep 2013 01:08:46 +0000 Received: by mail-wg0-f44.google.com with SMTP id b12so4917413wgh.35 for ; Mon, 09 Sep 2013 18:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudsoftcorp.com; s=google; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=mdM+AeqrtNXIWBpiQ1YHTJxZT6benZvckxPdNqN5ltM=; b=JbBKTrTaSwUF1QLTJpNg8G4B4X3GHZ8ieYCMGAHb2Wlte7AFVGrvfSQVcWgx2UTqCY xXiXzghfKYR4O3bFDtl5WWSXt2nj3ToSR8lROI9R4iMHmU+q3D+mNWP5aUnHBACZtuDP Raqw05p8E8F6t2j/Egphuas2ek3V8IG1qYTJc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type; bh=mdM+AeqrtNXIWBpiQ1YHTJxZT6benZvckxPdNqN5ltM=; b=MAFgxFt0mVFMHQ/rXnCk3KeMOO0FQ5MaGwsH+dNVnCx3mR7HfwnNoM+RTKRsjhnIsR wA8TcrcmHpwqyINXo+2SHbzi9zwz+9MO4+5kc351zk1MDNsUrtE6EAK0xZfLu9NCKSH8 yYoHgY0iqS5EGqDye4oWpfW9IlN1N5KngfWIzb4/OW4DP1+OL+vu9dpaJr5H/civ7IFf +14+H7UGMAuk3YniDwhg1rxv431XXXYo/oucTUbmtwjdIY+Oc16MpCOXVqa/9Ct5e4NL r9ew55q6Mmzrtm0ivKO+uXBvxgmNhU5KqVsk5aPpge0Xo47ySIfnEXLqqJo3+eXen1OY VzSg== X-Gm-Message-State: ALoCoQmpqEdyfobV+++RecCKJJvSaW7nIgAQhKkECJ3xfWr5qgO5I+0H1ct3WVJZr5oqf/0JtqCG X-Received: by 10.194.202.230 with SMTP id kl6mr15362152wjc.9.1378775305886; Mon, 09 Sep 2013 18:08:25 -0700 (PDT) Received: from almacretin.local (host86-131-198-237.range86-131.btcentralplus.com. [86.131.198.237]) by mx.google.com with ESMTPSA id pn7sm166173wic.6.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 09 Sep 2013 18:08:25 -0700 (PDT) Message-ID: <522E7107.7060902@CloudsoftCorp.com> Date: Tue, 10 Sep 2013 02:08:23 +0100 From: Alex Heneveld User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: user@cassandra.apache.org CC: Robert Coli Subject: Re: SchemaDisagreementError when launching a new Cassandra (1.2.2) cluster ? References: <522DE44A.50602@CloudsoftCorp.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------040103020004020909090601" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------040103020004020909090601 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Robert, Many thanks. Yes, it looks like a bug in 1.2.2. So far (6 runs) v 1.2.9 is acting as I had expected. (BTW re schema, I'm not defining anything myself so it is just the default/empty schema for which I was getting disagreeing versions.) Can I confirm I'm following best practice: when starting my cluster, I pick 2 nodes as seed nodes, then set up and start all the nodes in parallel, all configured to use those 2 seed nodes. Is there any care recommended in the start-up order? The docs suggest it doesn't matter too much so long as there is enough information to discover all the nodes, but I've seen some surprising behaviour. Besides the buggy 1.2.2, even with 1.2.9 if you try to be smart and set seeds@node1=node2 and seeds@node2=node1 it blows up rather spectacularly -- [1] ! Cheers Alex [1] java.lang.RuntimeException: No other nodes seen! Unable to bootstrap.If you intended to start a single-node cluster, you should make sure your broadcast_address (or listen_address) is listed as a seed. Otherwise, you need to determine why the seed being contacted has no knowledge of the rest of the cluster. Usually, this can be solved by giving all nodes the same seed list. On 09/09/2013 17:51, Robert Coli wrote: > On Mon, Sep 9, 2013 at 8:07 AM, Alex Heneveld > > wrote: > > The problem occurs in about 1 in 4 launches when I start a 2-node > cluster, where the two machines are configured identically with > both nodes as the seeds (apart from the listen_address being > different). On the problematic launches, describing schema > versions immediately after start shows that the two nodes have > different schemas (reported at both nodes) and any attempt to work > with the nodes returns the SDE. This is before I attempt to do > anything to the cluster. After ~60s the nodes reconcile their > differences, report a single schema used at both nodes, and I can > use the cluster without problems. > > > How are you defining the schema? > > I have a workaround, which is to use just one node to seed this > initial set. When the set of seeds is cardinality 1, the problem > does not occur. However the advice is to use 2 seeds and have > them be the same across the cluster -- so I'd like to get to the > bottom of this! > > > Seed nodes "cannot" bootstrap [1], so if you have RF=N and all nodes > as seeds, I'm not surprised that you are experiencing weird behavior. > A node booting with itself as a seed typically just starts up as a > cluster of one. > > I am running Cassandra 1.2.2 running in Amazon, using Brooklyn > (brooklyn.io ) to start and manage it. I can > share test cases, cassandra.yaml, logs, etc -- but am starting > with the above summary in case anyone can point me in the right > direction from that. > > > Cassandra 1.2.2 has significant bugs. You should launch with 1.2.9. > > 1.2.9 also has some bootstrap vs. seed fixes which might help your case. > > =Rob > [1] https://issues.apache.org/jira/browse/CASSANDRA-5836 --------------040103020004020909090601 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit

Robert,

Many thanks.  Yes, it looks like a bug in 1.2.2.  So far (6 runs) v 1.2.9 is acting as I had expected.

(BTW re schema, I'm not defining anything myself so it is just the default/empty schema for which I was getting disagreeing versions.)

Can I confirm I'm following best practice:  when starting my cluster, I pick 2 nodes as seed nodes, then set up and start all the nodes in parallel, all configured to use those 2 seed nodes.

Is there any care recommended in the start-up order?

The docs suggest it doesn't matter too much so long as there is enough information to discover all the nodes, but I've seen some surprising behaviour.  Besides the buggy 1.2.2, even with 1.2.9 if you try to be smart and set seeds@node1=node2 and seeds@node2=node1 it blows up rather spectacularly -- [1] !

Cheers
Alex

[1]  java.lang.RuntimeException: No other nodes seen!  Unable to bootstrap.If you intended to start a single-node cluster, you should make sure your broadcast_address (or listen_address) is listed as a seed.  Otherwise, you need to determine why the seed being contacted has no knowledge of the rest of the cluster.  Usually, this can be solved by giving all nodes the same seed list.


On 09/09/2013 17:51, Robert Coli wrote:
On Mon, Sep 9, 2013 at 8:07 AM, Alex Heneveld <alex.heneveld@cloudsoftcorp.com> wrote:
The problem occurs in about 1 in 4 launches when I start a 2-node cluster, where the two machines are configured identically with both nodes as the seeds (apart from the listen_address being different). On the problematic launches, describing schema versions immediately after start shows that the two nodes have different schemas (reported at both nodes) and any attempt to work with the nodes returns the SDE.  This is before I attempt to do anything to the cluster.  After ~60s the nodes reconcile their differences, report a single schema used at both nodes, and I can use the cluster without problems.

How are you defining the schema?
 
I have a workaround, which is to use just one node to seed this initial set.  When the set of seeds is cardinality 1, the problem does not occur.  However the advice is to use 2 seeds and have them be the same across the cluster -- so I'd like to get to the bottom of this!

Seed nodes "cannot" bootstrap [1], so if you have RF=N and all nodes as seeds, I'm not surprised that you are experiencing weird behavior. A node booting with itself as a seed typically just starts up as a cluster of one.
 
I am running Cassandra 1.2.2 running in Amazon, using Brooklyn (brooklyn.io) to start and manage it.  I can share test cases, cassandra.yaml, logs, etc -- but am starting with the above summary in case anyone can point me in the right direction from that.

Cassandra 1.2.2 has significant bugs. You should launch with 1.2.9.

1.2.9 also has some bootstrap vs. seed fixes which might help your case.

=Rob

--------------040103020004020909090601--