Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 86ABED52E for ; Wed, 5 Sep 2012 17:43:17 +0000 (UTC) Received: (qmail 5533 invoked by uid 500); 5 Sep 2012 17:43:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 5511 invoked by uid 500); 5 Sep 2012 17:43:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 5501 invoked by uid 99); 5 Sep 2012 17:43:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 17:43:14 +0000 X-ASF-Spam-Status: No, hits=2.6 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tyler@datastax.com designates 209.85.220.172 as permitted sender) Received: from [209.85.220.172] (HELO mail-vc0-f172.google.com) (209.85.220.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 17:43:10 +0000 Received: by vcbfo14 with SMTP id fo14so1314917vcb.31 for ; Wed, 05 Sep 2012 10:42:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=C4DcnsiRcZSCODVifPcXjLKX9iSRm6TI1EUn+9xOQeg=; b=C9pAVgPPvCBACKyYmTqiCITOh1OdlsxPaEpv0keHp+H1yHSuIV3nw8EO/17OyBn7H4 JHGMV7S49Qo0P3/sBGaNESGOafYO8vNsATn+MutrnXP0wtXj+/xQO2eqaVv9qG+97PCw p5wzpAPTlYhfzna44q7pw2TRHdWYMo5gBITcY2xySuLbt8+BFHAV/I9Shvyhrs/zV53d hhnceZbOaI8XZ72oKzuHrnZ427bMmdGSe3HNtdj9cdmtZoncgfeGDwyCuUJbdcfX+wEk FB2qGiK8DDZc+a58xVTY+XdJrphVulmRPVAws8fuEuiaVDbmrfIrUl0K4KpF3OfxT00f 9JRA== MIME-Version: 1.0 Received: by 10.220.150.16 with SMTP id w16mr3329177vcv.65.1346866968912; Wed, 05 Sep 2012 10:42:48 -0700 (PDT) Received: by 10.59.0.102 with HTTP; Wed, 5 Sep 2012 10:42:48 -0700 (PDT) In-Reply-To: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E658395@FISH-MBX-02.azaleos.net> References: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E637337@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E637B3A@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63928D@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E6397BB@FISH-MBX-02.azaleos.net> <4054967D-7C34-4964-A0AB-E9A0E59A2237@thelastpickle.com> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63B105@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63B493@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63BF90@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13@FISH-MBX-02.azaleos.net> <4161BDED-7E04-4E9D-953C-9048EC130954@thelastpickle.com> <6231C2AA-BC32-44AF-9E0E-1AB3065A256D@thelastpickle.com> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E6544F4@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E658395@FISH-MBX-02.azaleos.net> Date: Wed, 5 Sep 2012 12:42:48 -0500 Message-ID: Subject: Re: Expanding cluster to include a new DR datacenter From: Tyler Hobbs To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d043890953de4f504c8f7e752 X-Gm-Message-State: ALoCoQmvPFSgseQDKTzXqUgLeack8006qST/a72CuK8B8vDV0DvMKphk+E4rB7leRrhXXsgbl/Hq X-Virus-Checked: Checked by ClamAV on apache.org --f46d043890953de4f504c8f7e752 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I believe the issue should be resolved by this ticket, which should get into 1.1.5: https://issues.apache.org/jira/browse/CASSANDRA-4561 On Wed, Sep 5, 2012 at 12:33 PM, Bryce Godfrey w= rote: > I just wanted to raise this back up again, see if anyone can provide > some help. Thanks.**** > > ** ** > > *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > *Sent:* Friday, August 31, 2012 11:47 AM > > *To:* user@cassandra.apache.org > *Subject:* RE: Expanding cluster to include a new DR datacenter**** > > ** ** > > Here is the log around the drop keyspace with debugging enabled.**** > > ** ** > > *From:* aaron morton [mailto:aaron@thelastpickle.com] > > *Sent:* Wednesday, August 29, 2012 10:22 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > ** ** > > Can you provide the server log ? **** > > ** ** > > If you can turn the log level up to DEBUG that would be handy as well. **= * > * > > ** ** > > Cheers**** > > ** ** > > -----------------**** > > Aaron Morton**** > > Freelance Developer**** > > @aaronmorton**** > > http://www.thelastpickle.com**** > > ** ** > > On 30/08/2012, at 8:22 AM, Bryce Godfrey > wrote:**** > > ** ** > > Well I tried to drop the keyspace, but it=92s still there. No errors in > logs and Cassandra-cli showed the schema agreement after the command. I > took a snapshot of the system keyspace first. Nothing is crashing in the > clients yet either, still able to read/write to that keyspace.**** > > **** > > [default@EBonding] drop keyspace EBonding;**** > > 2eb11095-b8a8-31cd-80c3-c748d32a4208**** > > Waiting for schema agreement...**** > > ... schemas agree across the cluster**** > > **** > > [default@unknown] use EBonding;**** > > Authenticated to keyspace: EBonding**** > > [default@EBonding] describe;**** > > Keyspace: EBonding:**** > > Replication Strategy: org.apache.cassandra.locator.SimpleStrategy**** > > Durable Writes: true**** > > Options: [replication_factor:2]**** > > **** > > *From:* aaron morton [mailto:aaron@thelastpickle.com] > *Sent:* Wednesday, August 29, 2012 2:36 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > It would be handy to work out what the corruption is. Could you snapshot > the system keyspace and store it somewhere, just incase we can look at it > later ?**** > > **** > > Is there a way I can confirm this **** > > Errors in the client and/or the server log is the the traditional way. *= * > ** > > **** > > go about cleaning up/restoring the proper schema?**** > > If you need to get it back, and can handle the down time, the simple > thing is drop the KS and re-create it. Remember to take a snapshot first. > Drop keyspace takes one but it's the sort of thing I would do myself.**** > > **** > > Or you can try _try_ nodetool resetlocalschema. Without knowing what the > error is it's hard to say if it would work. **** > > **** > > Cheers**** > > **** > > **** > > -----------------**** > > Aaron Morton**** > > Freelance Developer**** > > @aaronmorton**** > > http://www.thelastpickle.com**** > > **** > > On 29/08/2012, at 9:10 AM, Bryce Godfrey > wrote:**** > > > > **** > > I believe what may be really going on is that my schema is in a bad or > corrupt state. I also have one keyspace that I just cannot drop an > existing column family from even though it shows no errors.**** > > **** > > So right now I was able to get 4 of my 6 keyspaces over to Network > Topology strategy.**** > > **** > > I think I got into this bad state after pointing Opscenter at this cluste= r > for the first time, as it started throwing errors after that and crashed = a > couple of my nodes until I stopped it and its agents.**** > > **** > > Is there a way I can confirm this or go about cleaning up/restoring the > proper schema?**** > > **** > > *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > *Sent:* Tuesday, August 28, 2012 11:09 AM > *To:* user@cassandra.apache.org > *Subject:* RE: Expanding cluster to include a new DR datacenter**** > > **** > > So in an interesting turn of events, this works on my other 4 keyspaces > but just not this =91EBonding=92 one which will not recognize the changes= . I > can probably get around this by dropping and re-creating this keyspace > since its uptime is not too important for us.**** > > **** > > [default@AlertStats] describe AlertStats;**** > > Keyspace: AlertStats:**** > > Replication Strategy: > org.apache.cassandra.locator.NetworkTopologyStrategy**** > > Durable Writes: true**** > > Options: [Fisher:3]**** > > **** > > *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com > ] > *Sent:* Monday, August 27, 2012 3:50 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > Can you describe your schema again with TierPoint in it?**** > > On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey > wrote:**** > > Same results. I restarted the node also to see if it just wasn=92t > picking up the changes and it still shows Simple. **** > > **** > > When I specify the DC for strategy_options I should be using the DC name > from properfy file snitch right? Ours is =93Fisher=94 and =93TierPoint= =94 so > that=92s what I used.**** > > **** > > *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com] > *Sent:* Monday, August 27, 2012 1:21 PM**** > > > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > In your update command is it possible to specify RF for both DC? You coul= d > just do DC1:2, DC2:0.**** > > On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey > wrote:**** > > Show schema output show the simple strategy still**** > > [default@unknown] show schema EBonding;**** > > create keyspace EBonding**** > > with placement_strategy =3D 'SimpleStrategy'**** > > and strategy_options =3D {replication_factor : 2}**** > > and durable_writes =3D true;**** > > **** > > This is the only thing I see in the system log at the time on all the > nodes:**** > > **** > > INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java > (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/22= 8 > serialized/live bytes, 4 ops)**** > > INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) > Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live > bytes, 4 ops)**** > > INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) > Completed flushing > /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34= 817-Data.db > (241 bytes) for commitlog p$**** > > **** > > **** > > Should I turn the logging level up on something to see some more info > maybe?**** > > **** > > *From:* aaron morton [mailto:aaron@thelastpickle.com] > *Sent:* Monday, August 27, 2012 1:35 AM**** > > > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > I did a quick test on a clean 1.1.4 and it worked **** > > **** > > Can you check the logs for errors ? Can you see your schema change in > there ?**** > > **** > > Also what is the output from show schema; in the cli ? **** > > **** > > Cheers**** > > **** > > -----------------**** > > Aaron Morton**** > > Freelance Developer**** > > @aaronmorton**** > > http://www.thelastpickle.com**** > > **** > > On 25/08/2012, at 6:53 PM, Bryce Godfrey > wrote:**** > > **** > > Yes**** > > **** > > [default@unknown] describe cluster;**** > > Cluster Information:**** > > Snitch: org.apache.cassandra.locator.PropertyFileSnitch**** > > Partitioner: org.apache.cassandra.dht.RandomPartitioner**** > > Schema versions:**** > > 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, > 10.20.8.1, 10.20.8.2, 10.20.8.3]**** > > **** > > *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com] > *Sent:* Friday, August 24, 2012 1:55 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > That's interesting can you do describe cluster?**** > > On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey > wrote:**** > > So I=92m at the point of updating the keyspaces from Simple to > NetworkTopology and I=92m not sure if the changes are being accepted usin= g > Cassandra-cli.**** > > **** > > I issue the change:**** > > **** > > [default@EBonding] update keyspace EBonding**** > > ... with placement_strategy =3D > 'org.apache.cassandra.locator.NetworkTopologyStrategy'**** > > ... and strategy_options=3D{Fisher:2};**** > > 9511e292-f1b6-3f78-b781-4c90aeb6b0f6**** > > Waiting for schema agreement...**** > > ... schemas agree across the cluster**** > > **** > > Then I do a describe and it still shows the old strategy. Is there > something else that I need to do? I=92ve exited and restarted Cassandra-= cli > and it still shows the SimpleStrategy for that keyspace. Other nodes sho= w > the same information.**** > > **** > > [default@EBonding] describe EBonding;**** > > Keyspace: EBonding:**** > > Replication Strategy: org.apache.cassandra.locator.SimpleStrategy**** > > Durable Writes: true**** > > Options: [replication_factor:2]**** > > **** > > **** > > *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > *Sent:* Thursday, August 23, 2012 11:06 AM > *To:* user@cassandra.apache.org > *Subject:* RE: Expanding cluster to include a new DR datacenter**** > > **** > > Thanks for the information! Answers my questions.**** > > **** > > *From:* Tyler Hobbs [mailto:tyler@datastax.com ] > *Sent:* Wednesday, August 22, 2012 7:10 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > If you didn't see this particular section, you may find it useful: > http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-d= ata-center-to-a-cluster > > Some comments inline:**** > > On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey > wrote:**** > > We are in the process of building out a new DR system in another Data > Center, and we want to mirror our Cassandra environment to that DR. I ha= ve > a couple questions on the best way to do this after reading the > documentation on the Datastax website. We didn=92t initially plan for th= is > to be a DR setup when first deployed a while ago due to budgeting, but no= w > we need to. So I=92m just trying to nail down the order of doing this as > well as any potential issues.**** > > **** > > For the nodes, we don=92t plan on querying the servers in this DR until w= e > fail over to this data center. We are going to have 5 similar nodes in > the DR, should I join them into the ring at token+1?**** > > > Join them at token+10 just to leave a little space. Make sure you're > using LOCAL_QUORUM for your queries instead of regular QUORUM. > **** > > **** > > All keyspaces are set to the replication strategy of SimpleStrategy. Can > I change the replication strategy after joining the new nodes in the DR t= o > NetworkTopologyStategy with the updated replication factor for each dr?**= * > * > > > Switch your keyspaces over to NetworkTopologyStrategy before adding the > new nodes. For the strategy options, just list the first dc until the > second is up (e.g. {main_dc: 3}). > **** > > **** > > Lastly, is changing snitch from default of SimpleSnitch to > RackInferringSnitch going to cause any issues? Since its in the > Cassandra.yaml file I assume a rolling restart to pick up the value would > be ok?**** > > > This is the first thing you'll want to do. Unless your node IPs would > naturally put all nodes in a DC in the same rack, I recommend using > PropertyFileSnitch, explicitly using the same rack. (I tend to prefer > PFSnitch regardless; it's harder to accidentally mess up.) A rolling > restart is required to pick up the change. Make sure to fill out > cassandra-topology.properties first if using PFSnitch. > **** > > **** > > This is all on Cassandra 1.1.4, Thanks for any help!**** > > **** > > **** > > > > > -- > Tyler Hobbs > DataStax **** > > **** > > **** > > ** ** > --=20 Tyler Hobbs DataStax --f46d043890953de4f504c8f7e752 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I believe the issue should be resolved by this ticket, which should get int= o 1.1.5: h= ttps://issues.apache.org/jira/browse/CASSANDRA-4561

On Wed, Sep 5, 2012 at 12:33 PM, Bryce Godfrey <Bryce.Godfrey@azal= eos.com> wrote:

I just wanted to raise th= is back up again, see if anyone can provide some help.=A0 Thanks.=

=A0<= /p>

From: Bryce = Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Friday, August 31, 2012 11:47 AM


To: u= ser@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter=

=A0

Here is the log around th= e drop keyspace with debugging enabled.

=A0<= /p>

From: aaron = morton [mailto= :aaron@thelastpickle.com]
Sent: Wednesday, August 29, 2012 10:22 PM
To: u= ser@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter=

=A0

Can you provide the server log ?=A0

=A0

If you can turn the log level up to DEBUG that would= be handy as well.=A0

=A0

Cheers

=A0

-----------------

Aaron Morton

Freelance Developer

@aaronmorton

=A0

On 30/08/2012, at 8:22 AM, Bryce Godfrey <Bryce.Godfrey@azale= os.com> wrote:

=A0

Well I tried to drop the = keyspace, but it=92s still there.=A0 No errors in logs and Cassandra-cli sh= owed the schema agreement after the command.=A0 I took a snapshot of the system keyspace first.=A0 Nothing is crashing in the clients yet ei= ther, still able to read/write to that keyspace.

=A0<= /p>

[default@EBonding] drop k= eyspace EBonding;

2eb11095-b8a8-31cd-80c3-c= 748d32a4208

Waiting for schema agreem= ent...

... schemas agree across = the cluster

=A0<= /p>

[default@unknown] use EBo= nding;

Authenticated to keyspace= : EBonding

[default@EBonding] descri= be;

Keyspace: EBonding:

=A0 Replication Strategy:= org.apache.cassandra.locator.SimpleStrategy

=A0 Durable Writes: true<= /span>

=A0=A0=A0 Options: [repli= cation_factor:2]

=A0<= /p>

From:= =A0aaron morton [mailto:aaron@thelastpickle.com]=A0
Sent:=A0Wednesday, August 29, 2012 2:36 AM
To:=A0user@cassandra.apache.org
Subject:=A0Re: Expanding cluster to include a new DR da= tacenter

=A0

It would be handy to work out what the corruption is= . Could you snapshot the system keyspace and store it somewhere, just incas= e we can look at it later ?

=A0

Is there a way I can conf= irm this=A0

Errors in the client and/or the server log is the th= e traditional way.=A0

=A0

go about cleaning up/rest= oring the proper schema?

If you need to get it back, and can handle the down = time, the simple thing is drop the KS and re-create it. Remember to take a = snapshot first. Drop keyspace takes one but it's the sort of thing I wo= uld do myself.

=A0

Or you can try _try_ nodetool resetlocalschema. =A0W= ithout knowing what the error is it's hard to say if it would work.=A0<= u>

=A0

Cheers

=A0

=A0

-----------------

Aaron Morton

Freelance Developer<= /u>

@aaronmorton

=A0

On 29/08/2012, at 9:10 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:



I believe what may be rea= lly going on is that my schema is in a bad or corrupt state.=A0 I also have= one keyspace that I just cannot drop an existing column family from even though it shows no errors.

=A0=A0

So right now I was able t= o get 4 of my 6 keyspaces over to Network Topology strategy.<= u>

=A0<= /p>

I think I got into this b= ad state after pointing Opscenter at this cluster for the first time, as it= started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.

=A0<= /p>

Is there a way I can conf= irm this or go about cleaning up/restoring the proper schema?=

=A0<= /p>

From:= =A0Bryce Godfrey [mailto:Bryce.= Godfrey@azaleos.com]=A0
Sent:=A0Tuesday, August 28, 2012 11:09 AM
To:=A0user@cassandra.apache.org=
Subject:=A0RE: Expanding cluster to include a new DR da= tacenter

=A0

So in an interesting turn= of events, this works on my other 4 keyspaces but just not this =91EBondin= g=92 one which will not recognize the changes.=A0 I can probably get around this by dropping and re-creating this keyspace since its uptime= is not too important for us.

=A0<= /p>

[default@AlertStats] desc= ribe AlertStats;

Keyspace: AlertStats:

=A0 Replication Strategy:= org.apache.cassandra.locator.NetworkTopologyStrategy<= /p>

=A0 Durable Writes: true<= /span>

=A0=A0=A0 Options: [Fishe= r:3]

=A0<= /p>

From:= =A0Mohit Anchlia [mailto:mohitanchlia@gmail.com]=A0<= /span>
Sent:=A0Monday, August 27, 2012 3:50 PM
To:=A0user@cassandra.apache.org=
Subject:=A0Re: Expanding cluster to include a new DR da= tacenter

=A0

Can you describe=A0yo= ur schema again with TierPoint in it?

On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <<= a href=3D"mailto:Bryce.Godfrey@azaleos.com" target=3D"_blank">Bryce.Godfrey@azaleos.com> wrote:=

Same results.=A0 I restar= ted the node also to see if it just wasn=92t picking up the changes and it = still shows Simple.=A0

=A0<= /p>

When I specify the DC for= strategy_options I should be using the DC name from properfy file snitch r= ight?=A0 Ours is =93Fisher=94 and =93TierPoint=94 so that=92s what I used.

=A0<= /p>

From:= =A0Mohit Anchlia [mailto:mohitanchlia@gmail.com]=A0<= /span>
Sent:=A0Monday, August 27, 2012 1:21 PM


To:=A0user@cassandra.apache.org=
Subject:=A0Re: Expanding cluster to include a new DR da= tacenter

=A0

In your update comman= d is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.=

On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <= Bryce.Godfrey@azaleos.com> wrote:=

Show schema output show t= he simple strategy still

[default@unknown] show sc= hema EBonding;

create keyspace EBonding<= /span>

=A0 with placement_strate= gy =3D 'SimpleStrategy'

=A0 and strategy_options = =3D {replication_factor : 2}

=A0 and durable_writes = =3D true;

=A0<= /p>

This is the only thing I = see in the system log at the time on all the nodes:

=A0<= /p>

INFO [MigrationStage:1] 2= 012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of= Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)

INFO [FlushWriter:765] 20= 12-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keys= paces@1157216346(183/228 serialized/live bytes, 4 ops)=

INFO [FlushWriter:765] 20= 12-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cass= andra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$

=A0<= /p>

=A0<= /p>

Should I turn the logging= level up on something to see some more info maybe?

=A0<= /p>

From:= =A0aaron morton [mailto:aaron@thelastpickle.com]=A0=
Sent:=A0Monday, August 27, 2012 1:35 AM


To:=A0user@cassandra.apache.org=
Subject:=A0Re: Expanding cluster to include a new DR da= tacenter

=A0

I did a quick test on a clean 1.1.4 and it worked=A0=

=A0

Can you check the logs for errors ? Can you see your= schema change in there ?

=A0

Also what is the output from show schema; in the cli= ?=A0

=A0

Cheers

=A0

-----------------

Aaron Morton

Freelance Developer<= /u>

@aaronmorton

=A0

On 25/08/2012, at 6:53 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:

=A0

Yes<= /p>

=A0<= /p>

[default@unknown] describ= e cluster;

Cluster Information:

=A0=A0 Snitch: org.apache= .cassandra.locator.PropertyFileSnitch

=A0=A0 Partitioner: org.a= pache.cassandra.dht.RandomPartitioner

=A0=A0 Schema versions:

=A0=A0=A0=A0=A0=A0=A0 951= 1e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.= 8.2, 10.20.8.3]

=A0<= /p>

From:=A0Mohi= t Anchlia [mailto:mohitanchlia@gmail.com]=A0 Sent:=A0Friday, August 24, 2012 1:55 PM
To:=A0user@cassandra.apache.org
Subject:=A0Re: Expanding cluster to include a new DR datacenter

=A0

That's interestin= g can you do describe cluster?

On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <= Bryce.Godfrey@azaleos.com> wrote:=

So I=92m at the point of = updating the keyspaces from Simple to NetworkTopology and I=92m not sure if= the changes are being accepted using Cassandra-cli.

=A0<= /p>

I issue the change:

=A0<= /p>

[default@EBonding] update= keyspace EBonding

...=A0=A0=A0=A0 with plac= ement_strategy =3D 'org.apache.cassandra.locator.NetworkTopologyStrateg= y'

...=A0=A0=A0=A0 and strat= egy_options=3D{Fisher:2};

9511e292-f1b6-3f78-b781-4= c90aeb6b0f6

Waiting for schema agreem= ent...

... schemas agree across = the cluster

=A0<= /p>

Then I do a describe and = it still shows the old strategy.=A0 Is there something else that I need to = do?=A0 I=92ve exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace.=A0 Other nodes show the same i= nformation.

=A0<= /p>

[default@EBonding] descri= be EBonding;

Keyspace: EBonding:

=A0 Replication Strategy:= org.apache.cassandra.locator.SimpleStrategy

=A0 Durable Writes: true<= /span>

=A0=A0=A0 Options: [repli= cation_factor:2]

=A0<= /p>

=A0<= /p>

From:=A0Bryc= e Godfrey [mailto:Bryce.Godfrey@azaleos.com]=A0=
Sent:=A0Thursday, August 23, 2012 11:06 AM
To:=A0user@cassandra.apache.org
Subject:=A0RE: Expanding cluster to include a new DR datacenter

=A0

Thanks for the informatio= n!=A0 Answers my questions.

=A0<= /p>

From:=A0Tyle= r Hobbs [mailto:tyler@datastax.com]=A0
Sent:=A0Wednesday, August 22, 2012 7:10 PM
To:=A0user@cassandra.apache.org
Subject:=A0Re: Expanding cluster to include a new DR datacenter

=A0

If you didn't see= this particular section, you may find it useful:http://www.datastax.= com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluste= r

Some comments inline:

On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <<= a href=3D"mailto:Bryce.Godfrey@azaleos.com" target=3D"_blank">Bryce.Godfrey@azaleos.com> wrote:=

We are in the process of building out a new DR syste= m in another Data Center, and we want to mirror our Cassandra environment t= o that DR.=A0 I have a couple questions on the best way to do this after re= ading the documentation on the Datastax website.=A0 We didn=92t initially plan for this to be a DR setup when firs= t deployed a while ago due to budgeting, but now we need to.=A0 So I=92m ju= st trying to nail down the order of doing this as well as any potential iss= ues.

=A0

For the nodes, we don=92t plan on querying the serve= rs in this DR until we fail over to this data center.=A0=A0 We are going to= have 5 similar nodes in the DR, should I join them into the ring at token+= 1?


Join them at token+10 just to leave a little space.=A0 Make sure you're= using LOCAL_QUORUM for your queries instead of regular QUORUM.
=A0

=A0

All keyspaces are set to the replication strategy of= SimpleStrategy.=A0 Can I change the replication strategy after joining the= new nodes in the DR to NetworkTopologyStategy with the updated replication= factor for each dr?


Switch your keyspaces over to NetworkTopologyStrategy before adding the new= nodes.=A0 For the strategy options, just list the first dc until the secon= d is up (e.g. {main_dc: 3}).
=A0

=A0

Lastly, is changing snitch from default of SimpleSni= tch to RackInferringSnitch going to cause any issues?=A0 Since its in the C= assandra.yaml file I assume a rolling restart to pick up the value would be= ok?


This is the first thing you'll want to do.=A0 Unless your node IPs woul= d naturally put all nodes in a DC in the same rack, I recommend using Prope= rtyFileSnitch, explicitly using the same rack.=A0 (I tend to prefer PFSnitc= h regardless; it's harder to accidentally mess up.)=A0 A rolling restart is required to pick up the change.=A0 Make = sure to fill out cassandra-topology.properties first if using PFSnitch.
=A0

=A0

This is all on Cassandra 1.1.4, Thanks for any help!=

=A0

=A0




--=A0
Tyler Hobbs
DataStax

=A0

=A0

=A0




--
Tyler Hobbs
DataStax
<= br> --f46d043890953de4f504c8f7e752--