I believe the issue should be resolved by this ticket, which should get into 1.1.5: https://issues.apache.org/jira/browse/CASSANDRA-4561 On Wed, Sep 5, 2012 at 12:33 PM, Bryce Godfrey wrote: > I just wanted to raise this back up again, see if anyone can provide > some help. Thanks.**** > > ** ** > > *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > *Sent:* Friday, August 31, 2012 11:47 AM > > *To:* user@cassandra.apache.org > *Subject:* RE: Expanding cluster to include a new DR datacenter**** > > ** ** > > Here is the log around the drop keyspace with debugging enabled.**** > > ** ** > > *From:* aaron morton [mailto:aaron@thelastpickle.com] > > *Sent:* Wednesday, August 29, 2012 10:22 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > ** ** > > Can you provide the server log ? **** > > ** ** > > If you can turn the log level up to DEBUG that would be handy as well. *** > * > > ** ** > > Cheers**** > > ** ** > > -----------------**** > > Aaron Morton**** > > Freelance Developer**** > > @aaronmorton**** > > http://www.thelastpickle.com**** > > ** ** > > On 30/08/2012, at 8:22 AM, Bryce Godfrey > wrote:**** > > ** ** > > Well I tried to drop the keyspace, but it’s still there. No errors in > logs and Cassandra-cli showed the schema agreement after the command. I > took a snapshot of the system keyspace first. Nothing is crashing in the > clients yet either, still able to read/write to that keyspace.**** > > **** > > [default@EBonding] drop keyspace EBonding;**** > > 2eb11095-b8a8-31cd-80c3-c748d32a4208**** > > Waiting for schema agreement...**** > > ... schemas agree across the cluster**** > > **** > > [default@unknown] use EBonding;**** > > Authenticated to keyspace: EBonding**** > > [default@EBonding] describe;**** > > Keyspace: EBonding:**** > > Replication Strategy: org.apache.cassandra.locator.SimpleStrategy**** > > Durable Writes: true**** > > Options: [replication_factor:2]**** > > **** > > *From:* aaron morton [mailto:aaron@thelastpickle.com] > *Sent:* Wednesday, August 29, 2012 2:36 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > It would be handy to work out what the corruption is. Could you snapshot > the system keyspace and store it somewhere, just incase we can look at it > later ?**** > > **** > > Is there a way I can confirm this **** > > Errors in the client and/or the server log is the the traditional way. ** > ** > > **** > > go about cleaning up/restoring the proper schema?**** > > If you need to get it back, and can handle the down time, the simple > thing is drop the KS and re-create it. Remember to take a snapshot first. > Drop keyspace takes one but it's the sort of thing I would do myself.**** > > **** > > Or you can try _try_ nodetool resetlocalschema. Without knowing what the > error is it's hard to say if it would work. **** > > **** > > Cheers**** > > **** > > **** > > -----------------**** > > Aaron Morton**** > > Freelance Developer**** > > @aaronmorton**** > > http://www.thelastpickle.com**** > > **** > > On 29/08/2012, at 9:10 AM, Bryce Godfrey > wrote:**** > > > > **** > > I believe what may be really going on is that my schema is in a bad or > corrupt state. I also have one keyspace that I just cannot drop an > existing column family from even though it shows no errors.**** > > **** > > So right now I was able to get 4 of my 6 keyspaces over to Network > Topology strategy.**** > > **** > > I think I got into this bad state after pointing Opscenter at this cluster > for the first time, as it started throwing errors after that and crashed a > couple of my nodes until I stopped it and its agents.**** > > **** > > Is there a way I can confirm this or go about cleaning up/restoring the > proper schema?**** > > **** > > *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > *Sent:* Tuesday, August 28, 2012 11:09 AM > *To:* user@cassandra.apache.org > *Subject:* RE: Expanding cluster to include a new DR datacenter**** > > **** > > So in an interesting turn of events, this works on my other 4 keyspaces > but just not this ‘EBonding’ one which will not recognize the changes. I > can probably get around this by dropping and re-creating this keyspace > since its uptime is not too important for us.**** > > **** > > [default@AlertStats] describe AlertStats;**** > > Keyspace: AlertStats:**** > > Replication Strategy: > org.apache.cassandra.locator.NetworkTopologyStrategy**** > > Durable Writes: true**** > > Options: [Fisher:3]**** > > **** > > *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com > ] > *Sent:* Monday, August 27, 2012 3:50 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > Can you describe your schema again with TierPoint in it?**** > > On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey > wrote:**** > > Same results. I restarted the node also to see if it just wasn’t > picking up the changes and it still shows Simple. **** > > **** > > When I specify the DC for strategy_options I should be using the DC name > from properfy file snitch right? Ours is “Fisher” and “TierPoint” so > that’s what I used.**** > > **** > > *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com] > *Sent:* Monday, August 27, 2012 1:21 PM**** > > > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > In your update command is it possible to specify RF for both DC? You could > just do DC1:2, DC2:0.**** > > On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey > wrote:**** > > Show schema output show the simple strategy still**** > > [default@unknown] show schema EBonding;**** > > create keyspace EBonding**** > > with placement_strategy = 'SimpleStrategy'**** > > and strategy_options = {replication_factor : 2}**** > > and durable_writes = true;**** > > **** > > This is the only thing I see in the system log at the time on all the > nodes:**** > > **** > > INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java > (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 > serialized/live bytes, 4 ops)**** > > INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) > Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live > bytes, 4 ops)**** > > INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) > Completed flushing > /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db > (241 bytes) for commitlog p$**** > > **** > > **** > > Should I turn the logging level up on something to see some more info > maybe?**** > > **** > > *From:* aaron morton [mailto:aaron@thelastpickle.com] > *Sent:* Monday, August 27, 2012 1:35 AM**** > > > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > I did a quick test on a clean 1.1.4 and it worked **** > > **** > > Can you check the logs for errors ? Can you see your schema change in > there ?**** > > **** > > Also what is the output from show schema; in the cli ? **** > > **** > > Cheers**** > > **** > > -----------------**** > > Aaron Morton**** > > Freelance Developer**** > > @aaronmorton**** > > http://www.thelastpickle.com**** > > **** > > On 25/08/2012, at 6:53 PM, Bryce Godfrey > wrote:**** > > **** > > Yes**** > > **** > > [default@unknown] describe cluster;**** > > Cluster Information:**** > > Snitch: org.apache.cassandra.locator.PropertyFileSnitch**** > > Partitioner: org.apache.cassandra.dht.RandomPartitioner**** > > Schema versions:**** > > 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, > 10.20.8.1, 10.20.8.2, 10.20.8.3]**** > > **** > > *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com] > *Sent:* Friday, August 24, 2012 1:55 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > That's interesting can you do describe cluster?**** > > On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey > wrote:**** > > So I’m at the point of updating the keyspaces from Simple to > NetworkTopology and I’m not sure if the changes are being accepted using > Cassandra-cli.**** > > **** > > I issue the change:**** > > **** > > [default@EBonding] update keyspace EBonding**** > > ... with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy'**** > > ... and strategy_options={Fisher:2};**** > > 9511e292-f1b6-3f78-b781-4c90aeb6b0f6**** > > Waiting for schema agreement...**** > > ... schemas agree across the cluster**** > > **** > > Then I do a describe and it still shows the old strategy. Is there > something else that I need to do? I’ve exited and restarted Cassandra-cli > and it still shows the SimpleStrategy for that keyspace. Other nodes show > the same information.**** > > **** > > [default@EBonding] describe EBonding;**** > > Keyspace: EBonding:**** > > Replication Strategy: org.apache.cassandra.locator.SimpleStrategy**** > > Durable Writes: true**** > > Options: [replication_factor:2]**** > > **** > > **** > > *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > *Sent:* Thursday, August 23, 2012 11:06 AM > *To:* user@cassandra.apache.org > *Subject:* RE: Expanding cluster to include a new DR datacenter**** > > **** > > Thanks for the information! Answers my questions.**** > > **** > > *From:* Tyler Hobbs [mailto:tyler@datastax.com ] > *Sent:* Wednesday, August 22, 2012 7:10 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Expanding cluster to include a new DR datacenter**** > > **** > > If you didn't see this particular section, you may find it useful: > http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster > > Some comments inline:**** > > On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey > wrote:**** > > We are in the process of building out a new DR system in another Data > Center, and we want to mirror our Cassandra environment to that DR. I have > a couple questions on the best way to do this after reading the > documentation on the Datastax website. We didn’t initially plan for this > to be a DR setup when first deployed a while ago due to budgeting, but now > we need to. So I’m just trying to nail down the order of doing this as > well as any potential issues.**** > > **** > > For the nodes, we don’t plan on querying the servers in this DR until we > fail over to this data center. We are going to have 5 similar nodes in > the DR, should I join them into the ring at token+1?**** > > > Join them at token+10 just to leave a little space. Make sure you're > using LOCAL_QUORUM for your queries instead of regular QUORUM. > **** > > **** > > All keyspaces are set to the replication strategy of SimpleStrategy. Can > I change the replication strategy after joining the new nodes in the DR to > NetworkTopologyStategy with the updated replication factor for each dr?*** > * > > > Switch your keyspaces over to NetworkTopologyStrategy before adding the > new nodes. For the strategy options, just list the first dc until the > second is up (e.g. {main_dc: 3}). > **** > > **** > > Lastly, is changing snitch from default of SimpleSnitch to > RackInferringSnitch going to cause any issues? Since its in the > Cassandra.yaml file I assume a rolling restart to pick up the value would > be ok?**** > > > This is the first thing you'll want to do. Unless your node IPs would > naturally put all nodes in a DC in the same rack, I recommend using > PropertyFileSnitch, explicitly using the same rack. (I tend to prefer > PFSnitch regardless; it's harder to accidentally mess up.) A rolling > restart is required to pick up the change. Make sure to fill out > cassandra-topology.properties first if using PFSnitch. > **** > > **** > > This is all on Cassandra 1.1.4, Thanks for any help!**** > > **** > > **** > > > > > -- > Tyler Hobbs > DataStax **** > > **** > > **** > > ** ** > -- Tyler Hobbs DataStax