From user-return-19414-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Aug 2 21:43:53 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7E79A61B3 for ; Tue, 2 Aug 2011 21:43:53 +0000 (UTC) Received: (qmail 27819 invoked by uid 500); 2 Aug 2011 21:43:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 27740 invoked by uid 500); 2 Aug 2011 21:43:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 27732 invoked by uid 99); 2 Aug 2011 21:43:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2011 21:43:50 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cyberaishu@gmail.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2011 21:43:46 +0000 Received: by fxe6 with SMTP id 6so242460fxe.31 for ; Tue, 02 Aug 2011 14:43:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=X6Loi/yCuwqBr7q1gIQ9bETeAtme/rDHKe6+wrZGF4o=; b=li743xd8SulzziA1Kcxt0sMqdTe7wIWygC3DMPsHbVP/8eFKKxNtTUd33U9W0UQRsr juH4pNlh7cWVfUOjv979C21GvcL1e8DDZLJFBk4XKdnfg+HAkDf/XDAdbC/20hBJL7+q v71Y5PSeqMeh2n9OtfWBfb8mMqB6zSchZTpGs= MIME-Version: 1.0 Received: by 10.204.143.25 with SMTP id s25mr1822034bku.205.1312321404349; Tue, 02 Aug 2011 14:43:24 -0700 (PDT) Received: by 10.204.52.212 with HTTP; Tue, 2 Aug 2011 14:43:24 -0700 (PDT) In-Reply-To: References: <1312320093.4058.7.camel@us-wash-ch2ljq1.morningstar.com> Date: Tue, 2 Aug 2011 14:43:24 -0700 Message-ID: Subject: Re: Nodetool ring not showing all nodes in cluster From: Aishwarya Venkataraman To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I corrected the seed list and checked the cluster name. They are all good now. But still nodetool -ring shows only one node. INFO 21:36:59,735 Starting Messaging Service on port 7000 INFO 21:36:59,748 Using saved token 11342745564031281485796955865106245222= 4 Nodes a_ipadrr and b_ipaddr have the same token 113427455640312814857969558651062452224. a_ipadrr is the new owner. All the nodes seem to be using the same initial token, despite me specifying an initial_token in the config file. Is this an issue ? How do I force cassandra to use the token in the cassandra.yaml file ? Thanks, Aishwarya On Tue, Aug 2, 2011 at 2:34 PM, Jonathan Ellis wrote: > Yes. > > Different cluster names could also cause this. > > On Tue, Aug 2, 2011 at 4:21 PM, Jeremiah Jordan > wrote: >> All of the nodes should have the same seedlist. =A0Don't use localhost a= s >> one of the items in it if you have multiple nodes. >> >> On Tue, 2011-08-02 at 10:10 -0700, Aishwarya Venkataraman wrote: >>> Nodetool does not show me all the nodes. Assuming I have three nodes >>> A, B and C. The seedlist of A is localhost. Seedlist of B is >>> localhost, A_ipaddr and seedlist of C is localhost,B_ipaddr,A_ipaddr. >>> I have autobootstrap set to false for all 3 nodes since they all have >>> the correct data and do not hav to migrate data from any particular >>> node. >>> >>> My problem here is why does n't nodetool ring show me all nodes in the >>> ring ? I agree that the cluster thinks that only one node is present. >>> How do I fix this ? >>> >>> Thanks, >>> Aishwarya >>> >>> >>> On Tue, Aug 2, 2011 at 9:56 AM, samal wrote: >>> > >>> > >>> >>> "ERROR 08:53:47,678 Internal error processing batch_mutate >>> >>> java.lang.IllegalStateException: replication factor (3) exceeds num= ber >>> >>> of endpoints (1)" >>> >> >>> >> You already answered >>> >> "It always keeps showing only one node and mentions that it is handl= ing >>> >> 100% of the load." >>> > >>> > Cluster think only one node is present in ring, it doesn't agree RF= =3D3 =A0it is >>> > expecting RF=3D1. >>> > Original Q: I m not exactly sure what is the problem. But >>> > Does nodetool ring show all the host? >>> > What is your seed list? >>> > Is bootstrapped node has seed ip of its own? >>> > AFAIK gossip work even without actively joining a ring. >>> > >>> >>> >>> >>> On Tue, Aug 2, 2011 at 7:21 AM, Aishwarya Venkataraman >>> >>> wrote: >>> >>> > Replies inline. >>> >>> > >>> >>> > Thanks, >>> >>> > Aishwarya >>> >>> > >>> >>> > On Tue, Aug 2, 2011 at 7:12 AM, Sorin Julean >>> >>> > wrote: >>> >>> >> Hi, >>> >>> >> >>> >>> >> =A0Until someone answers =A0with more details, few questions: >>> >>> >> =A01. did you moved the system keyspace as well ? >>> >>> > Yes. But I deleted the LocationInfo* files under the system folde= r. >>> >>> > Shall I go ahead and delete the entire system folder ? >>> >>> > >>> >>> >> =A02. the gossip IP of the new nodes are the same as the old one= s ? >>> >>> > No. The Ip is different. >>> >>> > >>> >>> >> =A03. which cassandra version are you running ? >>> >>> > I am using 0.8.1 >>> >>> > >>> >>> >> >>> >>> >> If 1. is yes and 2. is no, for a quick fix: take down the cluste= r, >>> >>> >> remove >>> >>> >> system keyspace, bring the cluster up and bootstrap the nodes. >>> >>> >> >>> >>> >> >>> >>> >> Kind regards, >>> >>> >> Sorin >>> >>> >> >>> >>> >> >>> >>> >> On Tue, Aug 2, 2011 at 2:53 PM, Aishwarya Venkataraman >>> >>> >> wrote: >>> >>> >>> >>> >>> >>> Hello, >>> >>> >>> >>> >>> >>> I recently migrated 400 GB of data that was on a different cass= andra >>> >>> >>> cluster (3 node with RF=3D 3) to a new cluster. I have a 3 node >>> >>> >>> =A0cluster >>> >>> >>> with replication factor set to three. When I run nodetool ring,= it >>> >>> >>> does not show me all the nodes in the cluster. It always keeps >>> >>> >>> showing >>> >>> >>> only one node and mentions that it is handling 100% of the load= . But >>> >>> >>> when I look at the logs, the nodes are able to talk to each oth= er via >>> >>> >>> the gossip protocol. Why does this happen ? Can you tell me wha= t I am >>> >>> >>> doing wrong ? >>> >>> >>> >>> >>> >>> Thanks, >>> >>> >>> Aishwarya >>> >>> >> >>> >>> >> >>> >>> > >>> >> >>> > >>> > >> >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >