Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0465DF84 for ; Tue, 28 Aug 2012 21:11:24 +0000 (UTC) Received: (qmail 59377 invoked by uid 500); 28 Aug 2012 21:11:22 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 59329 invoked by uid 500); 28 Aug 2012 21:11:22 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 59309 invoked by uid 99); 28 Aug 2012 21:11:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Aug 2012 21:11:22 +0000 X-ASF-Spam-Status: No, hits=1.0 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [216.82.249.131] (HELO mail28.messagelabs.com) (216.82.249.131) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Aug 2012 21:11:17 +0000 X-Env-Sender: Bryce.Godfrey@azaleos.com X-Msg-Ref: server-8.tower-28.messagelabs.com!1346188255!25723096!1 X-Originating-IP: [96.31.162.45] X-StarScan-Received: X-StarScan-Version: 6.6.1.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 15005 invoked from network); 28 Aug 2012 21:10:55 -0000 Received: from unknown (HELO smtp.azaleos.net) (96.31.162.45) by server-8.tower-28.messagelabs.com with AES128-SHA encrypted SMTP; 28 Aug 2012 21:10:55 -0000 Received: from FISH-MBX-02.azaleos.net ([fe80::4918:2597:a7be:1075]) by FISH-HUB-02.azaleos.net ([fe80::b18f:6491:3315:e104%11]) with mapi id 14.03.0083.000; Tue, 28 Aug 2012 14:10:55 -0700 From: Bryce Godfrey To: "user@cassandra.apache.org" Subject: RE: Expanding cluster to include a new DR datacenter Thread-Topic: Expanding cluster to include a new DR datacenter Thread-Index: Ac2ApIfFaOPYziBeTeqfUNwTaymK8QAapDIAABKmdlAANGtEMAAShFoAAAYy6sAAdtKxAAAFIMDQABOF5QAACnmiEP//1c6A//8y61D//jOKsA== Date: Tue, 28 Aug 2012 21:10:53 +0000 Message-ID: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13@FISH-MBX-02.azaleos.net> References: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E637337@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E637B3A@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63928D@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E6397BB@FISH-MBX-02.azaleos.net> <4054967D-7C34-4964-A0AB-E9A0E59A2237@thelastpickle.com> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63B105@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63B493@FISH-MBX-02.azaleos.net> <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63BF90@FISH-MBX-02.azaleos.net> In-Reply-To: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E63BF90@FISH-MBX-02.azaleos.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.0.100] Content-Type: multipart/alternative; boundary="_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13FISHMBX02azaleo_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13FISHMBX02azaleo_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I believe what may be really going on is that my schema is in a bad or corr= upt state. I also have one keyspace that I just cannot drop an existing co= lumn family from even though it shows no errors. So right now I was able to get 4 of my 6 keyspaces over to Network Topology= strategy. I think I got into this bad state after pointing Opscenter at this cluster = for the first time, as it started throwing errors after that and crashed a = couple of my nodes until I stopped it and its agents. Is there a way I can confirm this or go about cleaning up/restoring the pro= per schema? From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] Sent: Tuesday, August 28, 2012 11:09 AM To: user@cassandra.apache.org Subject: RE: Expanding cluster to include a new DR datacenter So in an interesting turn of events, this works on my other 4 keyspaces but= just not this 'EBonding' one which will not recognize the changes. I can = probably get around this by dropping and re-creating this keyspace since it= s uptime is not too important for us. [default@AlertStats] describe AlertStats; Keyspace: AlertStats: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrateg= y Durable Writes: true Options: [Fisher:3] From: Mohit Anchlia [mailto:mohitanchlia@gmail.com] Sent: Monday, August 27, 2012 3:50 PM To: user@cassandra.apache.org Subject: Re: Expanding cluster to include a new DR datacenter Can you describe your schema again with TierPoint in it? On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey > wrote: Same results. I restarted the node also to see if it just wasn't picking u= p the changes and it still shows Simple. When I specify the DC for strategy_options I should be using the DC name fr= om properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's = what I used. From: Mohit Anchlia [mailto:mohitanchlia@gmail.com] Sent: Monday, August 27, 2012 1:21 PM To: user@cassandra.apache.org Subject: Re: Expanding cluster to include a new DR datacenter In your update command is it possible to specify RF for both DC? You could = just do DC1:2, DC2:0. On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey > wrote: Show schema output show the simple strategy still [default@unknown] show schema EBonding; create keyspace EBonding with placement_strategy =3D 'SimpleStrategy' and strategy_options =3D {replication_factor : 2} and durable_writes =3D true; This is the only thing I see in the system log at the time on all the nodes= : INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (lin= e 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 seri= alized/live bytes, 4 ops) INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Wri= ting Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 = ops) INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Com= pleted flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_k= eyspaces-he-34817-Data.db (241 bytes) for commitlog p$ Should I turn the logging level up on something to see some more info maybe= ? From: aaron morton [mailto:aaron@thelastpickle.com] Sent: Monday, August 27, 2012 1:35 AM To: user@cassandra.apache.org Subject: Re: Expanding cluster to include a new DR datacenter I did a quick test on a clean 1.1.4 and it worked Can you check the logs for errors ? Can you see your schema change in there= ? Also what is the output from show schema; in the cli ? Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 25/08/2012, at 6:53 PM, Bryce Godfrey > wrote: Yes [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.PropertyFileSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.= 8.1, 10.20.8.2, 10.20.8.3] From: Mohit Anchlia [mailto:mohitanchlia@gmail.com] Sent: Friday, August 24, 2012 1:55 PM To: user@cassandra.apache.org Subject: Re: Expanding cluster to include a new DR datacenter That's interesting can you do describe cluster? On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey > wrote: So I'm at the point of updating the keyspaces from Simple to NetworkTopolog= y and I'm not sure if the changes are being accepted using Cassandra-cli. I issue the change: [default@EBonding] update keyspace EBonding ... with placement_strategy =3D 'org.apache.cassandra.locator.NetworkTo= pologyStrategy' ... and strategy_options=3D{Fisher:2}; 9511e292-f1b6-3f78-b781-4c90aeb6b0f6 Waiting for schema agreement... ... schemas agree across the cluster Then I do a describe and it still shows the old strategy. Is there somethi= ng else that I need to do? I've exited and restarted Cassandra-cli and it = still shows the SimpleStrategy for that keyspace. Other nodes show the sam= e information. [default@EBonding] describe EBonding; Keyspace: EBonding: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:2] From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] Sent: Thursday, August 23, 2012 11:06 AM To: user@cassandra.apache.org Subject: RE: Expanding cluster to include a new DR datacenter Thanks for the information! Answers my questions. From: Tyler Hobbs [mailto:tyler@datastax.com] Sent: Wednesday, August 22, 2012 7:10 PM To: user@cassandra.apache.org Subject: Re: Expanding cluster to include a new DR datacenter If you didn't see this particular section, you may find it useful:http://ww= w.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-= to-a-cluster Some comments inline: On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey > wrote: We are in the process of building out a new DR system in another Data Cente= r, and we want to mirror our Cassandra environment to that DR. I have a co= uple questions on the best way to do this after reading the documentation o= n the Datastax website. We didn't initially plan for this to be a DR setup= when first deployed a while ago due to budgeting, but now we need to. So = I'm just trying to nail down the order of doing this as well as any potenti= al issues. For the nodes, we don't plan on querying the servers in this DR until we fa= il over to this data center. We are going to have 5 similar nodes in the = DR, should I join them into the ring at token+1? Join them at token+10 just to leave a little space. Make sure you're using= LOCAL_QUORUM for your queries instead of regular QUORUM. All keyspaces are set to the replication strategy of SimpleStrategy. Can I= change the replication strategy after joining the new nodes in the DR to N= etworkTopologyStategy with the updated replication factor for each dr? Switch your keyspaces over to NetworkTopologyStrategy before adding the new= nodes. For the strategy options, just list the first dc until the second = is up (e.g. {main_dc: 3}). Lastly, is changing snitch from default of SimpleSnitch to RackInferringSni= tch going to cause any issues? Since its in the Cassandra.yaml file I assu= me a rolling restart to pick up the value would be ok? This is the first thing you'll want to do. Unless your node IPs would natu= rally put all nodes in a DC in the same rack, I recommend using PropertyFil= eSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regard= less; it's harder to accidentally mess up.) A rolling restart is required = to pick up the change. Make sure to fill out cassandra-topology.properties= first if using PFSnitch. This is all on Cassandra 1.1.4, Thanks for any help! -- Tyler Hobbs DataStax --_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13FISHMBX02azaleo_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I believe what may be rea= lly going on is that my schema is in a bad or corrupt state.  I also h= ave one keyspace that I just cannot drop an existing column family from even though it shows no errors.

  

So right now I was able t= o get 4 of my 6 keyspaces over to Network Topology strategy.

 <= /p>

I think I got into this b= ad state after pointing Opscenter at this cluster for the first time, as it= started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.

 <= /p>

Is there a way I can conf= irm this or go about cleaning up/restoring the proper schema?

 <= /p>

From: Bryce = Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Tuesday, August 28, 2012 11:09 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter

 

So in an interesting turn= of events, this works on my other 4 keyspaces but just not this ‘EBo= nding’ one which will not recognize the changes.  I can probably get around this by dropping and re-creating this keyspace since its uptime= is not too important for us.

 <= /p>

[default@AlertStats] desc= ribe AlertStats;

Keyspace: AlertStats:

  Replication Strate= gy: org.apache.cassandra.locator.NetworkTopologyStrategy<= /p>

  Durable Writes: tr= ue

    Option= s: [Fisher:3]

 <= /p>

From: Mohit = Anchlia [mailto:mohitanchlia@gmai= l.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apac= he.org
Subject: Re: Expanding cluster to include a new DR datacenter

 

Can you describe = ;your schema again with TierPoint in it?

On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <<= a href=3D"mailto:Bryce.Godfrey@azaleos.com" target=3D"_blank">Bryce.Godfrey= @azaleos.com> wrote:

Same results.  I restarted the nod= e also to see if it just wasn’t picking up the changes and it still shows Simple. 

 

When I specify the DC for strategy_opti= ons I should be using the DC name from properfy file snitch right?  Ours is “Fisher” and “TierPoint” so t= hat’s what I used.

 

From: Mohit Anchlia [mailt= o:mohitanchlia@= gmail.com]
Sent: Monday, August 27, 2012 1:21 PM


To: u= ser@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter

 

In your update command is it possible to specify RF for both DC? You cou= ld just do DC1:2, DC2:0.

On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com<= /a>> wrote:

Show schema output show the simple stra= tegy still

[default@unknown] show schema EBonding;=

create keyspace EBonding

  with placement_strategy =3D 'Sim= pleStrategy'

  and strategy_options =3D {replic= ation_factor : 2}

  and durable_writes =3D true;

 

This is the only thing I see in the sys= tem log at the time on all the nodes:

 

INFO [MigrationStage:1] 2012-08-27 10:5= 4:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 o= ps)

INFO [FlushWriter:765] 2012-08-27 10:54= :18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@11572163= 46(183/228 serialized/live bytes, 4 ops)

INFO [FlushWriter:765] 2012-08-27 10:54= :18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/sys= tem/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$

 

 

Should I turn the logging level up on s= omething to see some more info maybe?

 

From: aaron morton [mailto= :aaron@thelast= pickle.com]
Sent: Monday, August 27, 2012 1:35 AM


To: u= ser@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter=

 

I did a quick test on a clean 1.1.4 and it worked =

 

Can you check the logs for errors ? Can you see your schema change= in there ?

 

Also what is the output from show schema; in the cli ? <= /o:p>

 

Cheers

 

-----------------

Aaron Morton

Freelance Developer

@aaronmorton

 

On 25/08/2012, at 6:53 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>= wrote:

 

Yes

 

[default@unknown] describe cluster;

Cluster Information:<= /p>

   Snitch: org.apache.cassand= ra.locator.PropertyFileSnitch

   Partitioner: org.apache.ca= ssandra.dht.RandomPartitioner

   Schema versions:

      &nb= sp; 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1,= 10.20.8.2, 10.20.8.3]

 

From: Mohit Anchlia [= mailto:mohitanchlia@= gmail.com] 
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter

 

That's interesting can you do describe cluster?

On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:

So I’m at the point of updating t= he keyspaces from Simple to NetworkTopology and I’m not sure if the changes are being accepted using Cassandra-cli.

 

I issue the change:

 

[default@EBonding] update keyspace EBon= ding

...     with placem= ent_strategy =3D 'org.apache.cassandra.locator.NetworkTopologyStrategy'

...     and strateg= y_options=3D{Fisher:2};

9511e292-f1b6-3f78-b781-4c90aeb6b0f6

Waiting for schema agreement...<= o:p>

... schemas agree across the cluster

 

Then I do a describe and it still shows= the old strategy.  Is there something else that I need to do?  I’ve exited and restarted Cassandra-cli and it still shows= the SimpleStrategy for that keyspace.  Other nodes show the same info= rmation.

 

[default@EBonding] describe EBonding;

Keyspace: EBonding:

  Replication Strategy: org.apache= .cassandra.locator.SimpleStrategy

  Durable Writes: true=

    Options: [replicatio= n_factor:2]

 

 

From: Bryce Godfrey [= mailto:Bryce.Godfrey@azaleos.com] 
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter

 

Thanks for the information!  Answe= rs my questions.

 

From: Tyler Hobbs [mailto:tyler@datastax.com] 
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter

 

If you didn't see this particular section, you may find it useful:= http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-dat= a-center-to-a-cluster

Some comments inline:

On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:

We are in the process of building out a new DR system in another D= ata Center, and we want to mirror our Cassandra environment to that DR.&nbs= p; I have a couple questions on the best way to do this after reading the documentation on the Datastax website.&nb= sp; We didn’t initially plan for this to be a DR setup when first dep= loyed a while ago due to budgeting, but now we need to.  So I’m = just trying to nail down the order of doing this as well as any potential issues.

 

For the nodes, we don’t plan on querying the servers in this= DR until we fail over to this data center.   We are going to hav= e 5 similar nodes in the DR, should I join them into the ring at token+1?


Join them at token+10 just to leave a little space.  Make sure you= 're using LOCAL_QUORUM for your queries instead of regular QUORUM.
 

 

All keyspaces are set to the replication strategy of SimpleStrateg= y.  Can I change the replication strategy after joining the new nodes = in the DR to NetworkTopologyStategy with the updated replication factor for each dr?


Switch your keyspaces over to NetworkTopologyStrategy before adding the new= nodes.  For the strategy options, just list the first dc until the se= cond is up (e.g. {main_dc: 3}).
 

 

Lastly, is changing snitch from default of SimpleSnitch to RackInf= erringSnitch going to cause any issues?  Since its in the Cassandra.ya= ml file I assume a rolling restart to pick up the value would be ok?


This is the first thing you'll want to do.  Unless your node IPs would= naturally put all nodes in a DC in the same rack, I recommend using Proper= tyFileSnitch, explicitly using the same rack.  (I tend to prefer PFSni= tch regardless; it's harder to accidentally mess up.)  A rolling restart is required to pick up the change. = Make sure to fill out cassandra-topology.properties first if using PFSnitc= h.
 

 

This is all on Cassandra 1.1.4, Thanks for any help!

 

 




-- 
Tyler Hobbs
DataStax

 

 

 

--_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13FISHMBX02azaleo_--