cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <ty...@datastax.com>
Subject Re: Expanding cluster to include a new DR datacenter
Date Wed, 05 Sep 2012 17:42:48 GMT
I believe the issue should be resolved by this ticket, which should get
into 1.1.5: https://issues.apache.org/jira/browse/CASSANDRA-4561

On Wed, Sep 5, 2012 at 12:33 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>wrote:

>  I just wanted to raise this back up again, see if anyone can provide
> some help.  Thanks.****
>
> ** **
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Friday, August 31, 2012 11:47 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
>  ** **
>
> Here is the log around the drop keyspace with debugging enabled.****
>
> ** **
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com<aaron@thelastpickle.com>]
>
> *Sent:* Wednesday, August 29, 2012 10:22 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> Can you provide the server log ? ****
>
> ** **
>
> If you can turn the log level up to DEBUG that would be handy as well. ***
> *
>
> ** **
>
> Cheers****
>
> ** **
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ** **
>
> On 30/08/2012, at 8:22 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
> ** **
>
>  Well I tried to drop the keyspace, but it’s still there.  No errors in
> logs and Cassandra-cli showed the schema agreement after the command.  I
> took a snapshot of the system keyspace first.  Nothing is crashing in the
> clients yet either, still able to read/write to that keyspace.****
>
>  ****
>
> [default@EBonding] drop keyspace EBonding;****
>
> 2eb11095-b8a8-31cd-80c3-c748d32a4208****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
>  ****
>
> [default@unknown] use EBonding;****
>
> Authenticated to keyspace: EBonding****
>
> [default@EBonding] describe;****
>
> Keyspace: EBonding:****
>
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
>   Durable Writes: true****
>
>     Options: [replication_factor:2]****
>
>  ****
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Wednesday, August 29, 2012 2:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> It would be handy to work out what the corruption is. Could you snapshot
> the system keyspace and store it somewhere, just incase we can look at it
> later ?****
>
>  ****
>
>  Is there a way I can confirm this ****
>
>  Errors in the client and/or the server log is the the traditional way. **
> **
>
>  ****
>
>  go about cleaning up/restoring the proper schema?****
>
>  If you need to get it back, and can handle the down time, the simple
> thing is drop the KS and re-create it. Remember to take a snapshot first.
> Drop keyspace takes one but it's the sort of thing I would do myself.****
>
>  ****
>
> Or you can try _try_ nodetool resetlocalschema.  Without knowing what the
> error is it's hard to say if it would work. ****
>
>  ****
>
> Cheers****
>
>  ****
>
>  ****
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
>  ****
>
> On 29/08/2012, at 9:10 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
>
>
> ****
>
>  I believe what may be really going on is that my schema is in a bad or
> corrupt state.  I also have one keyspace that I just cannot drop an
> existing column family from even though it shows no errors.****
>
>   ****
>
> So right now I was able to get 4 of my 6 keyspaces over to Network
> Topology strategy.****
>
>  ****
>
> I think I got into this bad state after pointing Opscenter at this cluster
> for the first time, as it started throwing errors after that and crashed a
> couple of my nodes until I stopped it and its agents.****
>
>  ****
>
> Is there a way I can confirm this or go about cleaning up/restoring the
> proper schema?****
>
>  ****
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Tuesday, August 28, 2012 11:09 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> So in an interesting turn of events, this works on my other 4 keyspaces
> but just not this ‘EBonding’ one which will not recognize the changes.  I
> can probably get around this by dropping and re-creating this keyspace
> since its uptime is not too important for us.****
>
>  ****
>
> [default@AlertStats] describe AlertStats;****
>
> Keyspace: AlertStats:****
>
>   Replication Strategy:
> org.apache.cassandra.locator.NetworkTopologyStrategy****
>
>   Durable Writes: true****
>
>     Options: [Fisher:3]****
>
>  ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com<mohitanchlia@gmail.com>
> ]
> *Sent:* Monday, August 27, 2012 3:50 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> Can you describe your schema again with TierPoint in it?****
>
> On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
>  Same results.  I restarted the node also to see if it just wasn’t
> picking up the changes and it still shows Simple. ****
>
>  ****
>
> When I specify the DC for strategy_options I should be using the DC name
> from properfy file snitch right?  Ours is “Fisher” and “TierPoint” so
> that’s what I used.****
>
>  ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Monday, August 27, 2012 1:21 PM****
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> In your update command is it possible to specify RF for both DC? You could
> just do DC1:2, DC2:0.****
>
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
>  Show schema output show the simple strategy still****
>
> [default@unknown] show schema EBonding;****
>
> create keyspace EBonding****
>
>   with placement_strategy = 'SimpleStrategy'****
>
>   and strategy_options = {replication_factor : 2}****
>
>   and durable_writes = true;****
>
>  ****
>
> This is the only thing I see in the system log at the time on all the
> nodes:****
>
>  ****
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java
> (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228
> serialized/live bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264)
> Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live
> bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305)
> Completed flushing
> /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db
> (241 bytes) for commitlog p$****
>
>  ****
>
>  ****
>
> Should I turn the logging level up on something to see some more info
> maybe?****
>
>  ****
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Monday, August 27, 2012 1:35 AM****
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> I did a quick test on a clean 1.1.4 and it worked ****
>
>  ****
>
> Can you check the logs for errors ? Can you see your schema change in
> there ?****
>
>  ****
>
> Also what is the output from show schema; in the cli ? ****
>
>  ****
>
> Cheers****
>
>  ****
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
>  ****
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
>  ****
>
>  Yes****
>
>  ****
>
> [default@unknown] describe cluster;****
>
> Cluster Information:****
>
>    Snitch: org.apache.cassandra.locator.PropertyFileSnitch****
>
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner****
>
>    Schema versions:****
>
>         9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5,
> 10.20.8.1, 10.20.8.2, 10.20.8.3]****
>
>  ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Friday, August 24, 2012 1:55 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> That's interesting can you do describe cluster?****
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
>  So I’m at the point of updating the keyspaces from Simple to
> NetworkTopology and I’m not sure if the changes are being accepted using
> Cassandra-cli.****
>
>  ****
>
> I issue the change:****
>
>  ****
>
> [default@EBonding] update keyspace EBonding****
>
> ...     with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'****
>
> ...     and strategy_options={Fisher:2};****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
>  ****
>
> Then I do a describe and it still shows the old strategy.  Is there
> something else that I need to do?  I’ve exited and restarted Cassandra-cli
> and it still shows the SimpleStrategy for that keyspace.  Other nodes show
> the same information.****
>
>  ****
>
> [default@EBonding] describe EBonding;****
>
> Keyspace: EBonding:****
>
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
>   Durable Writes: true****
>
>     Options: [replication_factor:2]****
>
>  ****
>
>  ****
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Thursday, August 23, 2012 11:06 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> Thanks for the information!  Answers my questions.****
>
>  ****
>
> *From:* Tyler Hobbs [mailto:tyler@datastax.com <tyler@datastax.com>]
> *Sent:* Wednesday, August 22, 2012 7:10 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
>  ****
>
> If you didn't see this particular section, you may find it useful:
> http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:****
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com>
> wrote:****
>
>  We are in the process of building out a new DR system in another Data
> Center, and we want to mirror our Cassandra environment to that DR.  I have
> a couple questions on the best way to do this after reading the
> documentation on the Datastax website.  We didn’t initially plan for this
> to be a DR setup when first deployed a while ago due to budgeting, but now
> we need to.  So I’m just trying to nail down the order of doing this as
> well as any potential issues.****
>
>  ****
>
> For the nodes, we don’t plan on querying the servers in this DR until we
> fail over to this data center.   We are going to have 5 similar nodes in
> the DR, should I join them into the ring at token+1?****
>
>
> Join them at token+10 just to leave a little space.  Make sure you're
> using LOCAL_QUORUM for your queries instead of regular QUORUM.
>  ****
>
>   ****
>
> All keyspaces are set to the replication strategy of SimpleStrategy.  Can
> I change the replication strategy after joining the new nodes in the DR to
> NetworkTopologyStategy with the updated replication factor for each dr?***
> *
>
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the
> new nodes.  For the strategy options, just list the first dc until the
> second is up (e.g. {main_dc: 3}).
>  ****
>
>   ****
>
> Lastly, is changing snitch from default of SimpleSnitch to
> RackInferringSnitch going to cause any issues?  Since its in the
> Cassandra.yaml file I assume a rolling restart to pick up the value would
> be ok?****
>
>
> This is the first thing you'll want to do.  Unless your node IPs would
> naturally put all nodes in a DC in the same rack, I recommend using
> PropertyFileSnitch, explicitly using the same rack.  (I tend to prefer
> PFSnitch regardless; it's harder to accidentally mess up.)  A rolling
> restart is required to pick up the change.  Make sure to fill out
> cassandra-topology.properties first if using PFSnitch.
>  ****
>
>   ****
>
> This is all on Cassandra 1.1.4, Thanks for any help!****
>
>  ****
>
>  ****
>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>****
>
>     ****
>
>    ****
>
>     ** **
>



-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Mime
View raw message