cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Expanding cluster to include a new DR datacenter
Date Thu, 30 Aug 2012 05:22:09 GMT
Can you provide the server log ? 

If you can turn the log level up to DEBUG that would be handy as well. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 30/08/2012, at 8:22 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:

> Well I tried to drop the keyspace, but it’s still there.  No errors in logs and Cassandra-cli
showed the schema agreement after the command.  I took a snapshot of the system keyspace first.
 Nothing is crashing in the clients yet either, still able to read/write to that keyspace.
>  
> [default@EBonding] drop keyspace EBonding;
> 2eb11095-b8a8-31cd-80c3-c748d32a4208
> Waiting for schema agreement...
> ... schemas agree across the cluster
>  
> [default@unknown] use EBonding;
> Authenticated to keyspace: EBonding
> [default@EBonding] describe;
> Keyspace: EBonding:
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>   Durable Writes: true
>     Options: [replication_factor:2]
>  
> From: aaron morton [mailto:aaron@thelastpickle.com] 
> Sent: Wednesday, August 29, 2012 2:36 AM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>  
> It would be handy to work out what the corruption is. Could you snapshot the system keyspace
and store it somewhere, just incase we can look at it later ?
>  
> Is there a way I can confirm this 
> Errors in the client and/or the server log is the the traditional way. 
>  
> go about cleaning up/restoring the proper schema?
> If you need to get it back, and can handle the down time, the simple thing is drop the
KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the
sort of thing I would do myself.
>  
> Or you can try _try_ nodetool resetlocalschema.  Without knowing what the error is it's
hard to say if it would work. 
>  
> Cheers
>  
>  
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 29/08/2012, at 9:10 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> 
> 
> I believe what may be really going on is that my schema is in a bad or corrupt state.
 I also have one keyspace that I just cannot drop an existing column family from even though
it shows no errors.
>   
> So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
>  
> I think I got into this bad state after pointing Opscenter at this cluster for the first
time, as it started throwing errors after that and crashed a couple of my nodes until I stopped
it and its agents.
>  
> Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
>  
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] 
> Sent: Tuesday, August 28, 2012 11:09 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>  
> So in an interesting turn of events, this works on my other 4 keyspaces but just not
this ‘EBonding’ one which will not recognize the changes.  I can probably get around this
by dropping and re-creating this keyspace since its uptime is not too important for us.
>  
> [default@AlertStats] describe AlertStats;
> Keyspace: AlertStats:
>   Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
>   Durable Writes: true
>     Options: [Fisher:3]
>  
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com] 
> Sent: Monday, August 27, 2012 3:50 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>  
> Can you describe your schema again with TierPoint in it?
> 
> On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> Same results.  I restarted the node also to see if it just wasn’t picking up the changes
and it still shows Simple. 
>  
> When I specify the DC for strategy_options I should be using the DC name from properfy
file snitch right?  Ours is “Fisher” and “TierPoint” so that’s what I used.
>  
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com] 
> Sent: Monday, August 27, 2012 1:21 PM
> 
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>  
> In your update command is it possible to specify RF for both DC? You could just do DC1:2,
DC2:0.
> 
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> Show schema output show the simple strategy still
> [default@unknown] show schema EBonding;
> create keyspace EBonding
>   with placement_strategy = 'SimpleStrategy'
>   and strategy_options = {replication_factor : 2}
>   and durable_writes = true;
>  
> This is the only thing I see in the system log at the time on all the nodes:
>  
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228
serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing
/opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241
bytes) for commitlog p$
>  
>  
> Should I turn the logging level up on something to see some more info maybe?
>  
> From: aaron morton [mailto:aaron@thelastpickle.com] 
> Sent: Monday, August 27, 2012 1:35 AM
> 
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>  
> I did a quick test on a clean 1.1.4 and it worked 
>  
> Can you check the logs for errors ? Can you see your schema change in there ?
>  
> Also what is the output from show schema; in the cli ? 
>  
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
>  
> 
> Yes
>  
> [default@unknown] describe cluster;
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.PropertyFileSnitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions:
>         9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2,
10.20.8.3]
>  
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com] 
> Sent: Friday, August 24, 2012 1:55 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>  
> That's interesting can you do describe cluster?
> 
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> So I’m at the point of updating the keyspaces from Simple to NetworkTopology and I’m
not sure if the changes are being accepted using Cassandra-cli.
>  
> I issue the change:
>  
> [default@EBonding] update keyspace EBonding
> ...     with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
> ...     and strategy_options={Fisher:2};
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6
> Waiting for schema agreement...
> ... schemas agree across the cluster
>  
> Then I do a describe and it still shows the old strategy.  Is there something else that
I need to do?  I’ve exited and restarted Cassandra-cli and it still shows the SimpleStrategy
for that keyspace.  Other nodes show the same information.
>  
> [default@EBonding] describe EBonding;
> Keyspace: EBonding:
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>   Durable Writes: true
>     Options: [replication_factor:2]
>  
>  
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] 
> Sent: Thursday, August 23, 2012 11:06 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>  
> Thanks for the information!  Answers my questions.
>  
> From: Tyler Hobbs [mailto:tyler@datastax.com] 
> Sent: Wednesday, August 22, 2012 7:10 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>  
> If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
> 
> Some comments inline:
> 
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> We are in the process of building out a new DR system in another Data Center, and we
want to mirror our Cassandra environment to that DR.  I have a couple questions on the best
way to do this after reading the documentation on the Datastax website.  We didn’t initially
plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we
need to.  So I’m just trying to nail down the order of doing this as well as any potential
issues.
>  
> For the nodes, we don’t plan on querying the servers in this DR until we fail over
to this data center.   We are going to have 5 similar nodes in the DR, should I join them
into the ring at token+1?
> 
> Join them at token+10 just to leave a little space.  Make sure you're using LOCAL_QUORUM
for your queries instead of regular QUORUM.
>  
>  
> All keyspaces are set to the replication strategy of SimpleStrategy.  Can I change the
replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with
the updated replication factor for each dr?
> 
> Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes.  For
the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
>  
>  
> Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going
to cause any issues?  Since its in the Cassandra.yaml file I assume a rolling restart to pick
up the value would be ok?
> 
> This is the first thing you'll want to do.  Unless your node IPs would naturally put
all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using
the same rack.  (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.)
 A rolling restart is required to pick up the change.  Make sure to fill out cassandra-topology.properties
first if using PFSnitch.
>  
>  
> This is all on Cassandra 1.1.4, Thanks for any help!
>  
>  
> 
> 
> 
> -- 
> Tyler Hobbs
> DataStax
> 
>  
>  


Mime
View raw message