Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 404C29C93 for ; Mon, 3 Oct 2011 17:15:18 +0000 (UTC) Received: (qmail 28244 invoked by uid 500); 3 Oct 2011 17:15:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 28216 invoked by uid 500); 3 Oct 2011 17:15:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28208 invoked by uid 99); 3 Oct 2011 17:15:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Oct 2011 17:15:15 +0000 X-ASF-Spam-Status: No, hits=1.6 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of eczech52@gmail.com designates 209.85.216.44 as permitted sender) Received: from [209.85.216.44] (HELO mail-qw0-f44.google.com) (209.85.216.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Oct 2011 17:15:09 +0000 Received: by qadb12 with SMTP id b12so2313684qad.31 for ; Mon, 03 Oct 2011 10:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; bh=BIODQsfbsjOwfWMYp8M8Wk6s/yi2jgeIRezbLASRYvo=; b=dPKcReevba8eHgDDrb2BOk8bd+LjmOUDi9x8NsZsG5wX1T77Gt8S+4AMRozrzYY2Ic QIq+Okt/TOf2PceGlE+e3N+WabgREmBKQCMvVgolMVeA7XXbzZi+ahlCP2VYqg/ZBVyK 9UBj3LPi+rCvjbPTAn3qeGrvYph5IWmYbp2Gw= Received: by 10.43.44.134 with SMTP id ug6mr291356icb.25.1317662088265; Mon, 03 Oct 2011 10:14:48 -0700 (PDT) MIME-Version: 1.0 Sender: eczech52@gmail.com Received: by 10.42.222.135 with HTTP; Mon, 3 Oct 2011 10:14:26 -0700 (PDT) In-Reply-To: <87ty7qolj4.fsf@lambda.ihatejava.org> References: <87ty7r2hdw.fsf@lambda.ihatejava.org> <87pqif2g8x.fsf@lambda.ihatejava.org> <87ty7qolj4.fsf@lambda.ihatejava.org> From: Eric Czech Date: Mon, 3 Oct 2011 11:14:26 -0600 X-Google-Sender-Auth: DKlaKXm8d0_iJH1vzbvDjeX8LRQ Message-ID: Subject: Re: unwanted node discovery To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec52999ddb47f5904ae681ca3 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec52999ddb47f5904ae681ca3 Content-Type: text/plain; charset=ISO-8859-1 The tokens were different than the production cluster and after closer inspection a lot of data wasn't queryable (as excpected I suppose). I set the tokens and everything seems ok now. Auto bootstrap was false so no issues there. Thanks for the insight Shyamal! It's good to finally have this up and running. On Sun, Oct 2, 2011 at 8:29 PM, Shyamal Prasad wrote: > >>>>> "Eric" == Eric Czech writes: > > Eric> Yea that's not a mapping I'd like to maintain either -- as an > Eric> experiment, I copied production sstables to the analysis > Eric> cluster and ran brisk/cassandra without specifying an initial > Eric> token (after deleting the LocationInfo* files and renaming the > Eric> cluster). > > Based on my understanding this will allow everything to start up, yes. > > Eric> As far as I can tell, everything is running normally but I'm > Eric> not sure how the cluster chose tokens for the nodes given that > Eric> I didn't specify them after just dropping the raw sstables > Eric> in. I can still read data as usual from the column families > Eric> that were copied but I'm not sure how not specifying the > Eric> tokens affects everything. > > Did you check the ring to see what tokens you got for the analysis > cluster? I would be surprised if you got the same ring configuration as > production. > > Eric> Is some of my data just unreachable now because the tokens > Eric> weren't manually defined? > > I suspect your data is messed up. But the best way to determine it would > be to examine the ring (use nodetool) - if it is the same as your > production cluster you are good to go. > > Also, did you set your (non seed) nodes in the analysis cluster to auto > bootstrap or not? That impacts what happens. > > > Eric> This doesn't appear to be the case but is this something you > Eric> have tried too or do you understand the storage / topology > Eric> logic well enough to know that this isn't a viable strategy? > > No and No. I have been reading the code. Line 497 of > org.apache.cassandra.service.StorageService.java on trunk is a good > place to start since what happens depends somewhat on your specific > cassandra.yaml settings (specifically auto bootstrap). > > I would be betting you are getting random tokens (look for "Generated > random token..." in your log). Don't trust me, read the code. I have all > of two weeks of experience with this stuff (and it's not quite my day > job to be doing it either :-) > > Bottom line: I think you need to fix the seeds for your use case. > > Cheers! > Shyamal > --bcaec52999ddb47f5904ae681ca3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The tokens were different than the production cluster and after closer insp= ection a lot of data wasn't queryable (as excpected I suppose). =A0I se= t the tokens and everything seems ok now.

Auto bootstrap= was false so no issues there.=A0

Thanks for the insight Shyamal! =A0It's good to fin= ally have this up and running.


On Sun, Oct 2, 2011 at 8:29 PM, Shyamal Prasad <shyamal@member.fsf.org&= gt; wrote:
>>>>> &quo= t;Eric" =3D=3D Eric Czech <eric@nextbigsound.com> writes:

=A0 =A0Eric> Yea that's not a mapping I'd like to maintai= n either -- as an
=A0 =A0Eric> experiment, I copied production sstables to the analysis =A0 =A0Eric> cluster and ran brisk/cassandra without specifying an init= ial
=A0 =A0Eric> token (after deleting the LocationInfo* files and renaming= the
=A0 =A0Eric> cluster).

Based on my understanding this will allow everything to start up, yes.

=A0 =A0Eric> =A0As far as I can tell, everything is running normally bu= t I'm
=A0 =A0Eric> not sure how the cluster chose tokens for the nodes given = that
=A0 =A0Eric> I didn't specify them after just dropping the raw ssta= bles
=A0 =A0Eric> in. =A0I can still read data as usual from the column fami= lies
=A0 =A0Eric> that were copied but I'm not sure how not specifying t= he
=A0 =A0Eric> tokens affects everything.

Did you check the ring to see what tokens you got for the analysis
cluster? I would be surprised if you got the same ring configuration as
production.

=A0 =A0Eric> =A0Is some of my data just unreachable now because the tok= ens
=A0 =A0Eric> weren't manually defined?

I suspect your data is messed up. But the best way to determine it would be to examine the ring (use nodetool) - if it is the same as your
production cluster you are good to go.

Also, did you set your (non seed) nodes in the analysis cluster to auto
bootstrap or not? That impacts what happens.


=A0 =A0Eric> =A0This doesn't appear to be the case but is this some= thing you
=A0 =A0Eric> have tried too or do you understand the storage / topology=
=A0 =A0Eric> logic well enough to know that this isn't a viable str= ategy?

No and No. I have been reading the code. Line 497 of
org.apache.cassandra.service.StorageService.java on trunk is a good
place to start since what happens depends somewhat on your specific
cassandra.yaml settings (specifically auto bootstrap).

I would be betting you are getting random tokens (look for "Generated<= br> random token..." in your log). Don't trust me, read the code. I ha= ve all
of two weeks of experience with this stuff (and it's not quite my day job to be doing it either :-)

Bottom line: I think you need to fix the seeds for your use case.

Cheers!
Shyamal

--bcaec52999ddb47f5904ae681ca3--