Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 395BF747C for ; Tue, 6 Dec 2011 01:03:58 +0000 (UTC) Received: (qmail 33521 invoked by uid 500); 6 Dec 2011 01:03:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 33499 invoked by uid 500); 6 Dec 2011 01:03:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 33491 invoked by uid 99); 6 Dec 2011 01:03:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2011 01:03:56 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of potekhin@bnl.gov designates 130.199.3.132 as permitted sender) Received: from [130.199.3.132] (HELO smtpgw.bnl.gov) (130.199.3.132) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2011 01:03:47 +0000 X-BNL-policy-q: X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AiYFADtp3U6CxzYH/2dsb2JhbAA+BoUFow6CMYEFgXMBBSMVQBElAgUWCwICCQMCAQIBRRMIAQGrGZF/gTCGSBSBf4EWBIgtkXaMbw X-IronPort-AV: E=Sophos;i="4.71,302,1320642000"; d="scan'208";a="156282067" Received: from rcf.rhic.bnl.gov ([130.199.54.7]) by smtpgw.sec.bnl.local with ESMTP/TLS/DHE-RSA-AES256-SHA; 05 Dec 2011 20:03:26 -0500 Received: from [192.168.0.192] (ool-18bde93d.dyn.optonline.net [24.189.233.61]) (authenticated bits=0) by rcf.rhic.bnl.gov (8.13.8/8.13.8) with ESMTP id pB613Pam005787 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 5 Dec 2011 20:03:25 -0500 Message-ID: <4EDD69E6.7000802@bnl.gov> Date: Mon, 05 Dec 2011 20:03:34 -0500 From: Maxim Potekhin User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: Could not reach schema agreement... 0.8.6 References: <4EDAAF7E.40502@bnl.gov> <4EDACE10.6010804@bnl.gov> <4EDBB7A9.9060604@bnl.gov> <4EDBE255.4080905@bnl.gov> <4EDC0783.90509@bnl.gov> <4EDD1F0F.3020007@bnl.gov> In-Reply-To: <4EDD1F0F.3020007@bnl.gov> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hello, upon startup, in my cluster of 3 machines, I see similar messages in system.log on each node (below). I start nodes one by one, after I ascertain the previous one is online. So they can't reach schema agreement, all of them. Why? No unusual load visible in Ganglia plots. ERROR [HintedHandoff:1] 2011-12-05 19:52:17,426 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Hint edHandoff:1,1,main] java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /130.199.185.194 in 60000ms at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)