Can you provide the server log ?
If you can turn the log level up to DEBUG that would be handy as well.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 30/08/2012, at 8:22 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> Well I tried to drop the keyspace, but it’s still there. No errors in logs and Cassandra-cli
showed the schema agreement after the command. I took a snapshot of the system keyspace first.
Nothing is crashing in the clients yet either, still able to read/write to that keyspace.
>
> [default@EBonding] drop keyspace EBonding;
> 2eb11095-b8a8-31cd-80c3-c748d32a4208
> Waiting for schema agreement...
> ... schemas agree across the cluster
>
> [default@unknown] use EBonding;
> Authenticated to keyspace: EBonding
> [default@EBonding] describe;
> Keyspace: EBonding:
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> Durable Writes: true
> Options: [replication_factor:2]
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, August 29, 2012 2:36 AM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> It would be handy to work out what the corruption is. Could you snapshot the system keyspace
and store it somewhere, just incase we can look at it later ?
>
> Is there a way I can confirm this
> Errors in the client and/or the server log is the the traditional way.
>
> go about cleaning up/restoring the proper schema?
> If you need to get it back, and can handle the down time, the simple thing is drop the
KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the
sort of thing I would do myself.
>
> Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's
hard to say if it would work.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/08/2012, at 9:10 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
>
>
> I believe what may be really going on is that my schema is in a bad or corrupt state.
I also have one keyspace that I just cannot drop an existing column family from even though
it shows no errors.
>
> So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
>
> I think I got into this bad state after pointing Opscenter at this cluster for the first
time, as it started throwing errors after that and crashed a couple of my nodes until I stopped
it and its agents.
>
> Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Tuesday, August 28, 2012 11:09 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> So in an interesting turn of events, this works on my other 4 keyspaces but just not
this ‘EBonding’ one which will not recognize the changes. I can probably get around this
by dropping and re-creating this keyspace since its uptime is not too important for us.
>
> [default@AlertStats] describe AlertStats;
> Keyspace: AlertStats:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [Fisher:3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Monday, August 27, 2012 3:50 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> Can you describe your schema again with TierPoint in it?
>
> On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> Same results. I restarted the node also to see if it just wasn’t picking up the changes
and it still shows Simple.
>
> When I specify the DC for strategy_options I should be using the DC name from properfy
file snitch right? Ours is “Fisher” and “TierPoint” so that’s what I used.
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Monday, August 27, 2012 1:21 PM
>
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> In your update command is it possible to specify RF for both DC? You could just do DC1:2,
DC2:0.
>
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> Show schema output show the simple strategy still
> [default@unknown] show schema EBonding;
> create keyspace EBonding
> with placement_strategy = 'SimpleStrategy'
> and strategy_options = {replication_factor : 2}
> and durable_writes = true;
>
> This is the only thing I see in the system log at the time on all the nodes:
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228
serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing
/opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241
bytes) for commitlog p$
>
>
> Should I turn the logging level up on something to see some more info maybe?
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, August 27, 2012 1:35 AM
>
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> I did a quick test on a clean 1.1.4 and it worked
>
> Can you check the logs for errors ? Can you see your schema change in there ?
>
> Also what is the output from show schema; in the cli ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
>
>
> Yes
>
> [default@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2,
10.20.8.3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Friday, August 24, 2012 1:55 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> That's interesting can you do describe cluster?
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> So I’m at the point of updating the keyspaces from Simple to NetworkTopology and I’m
not sure if the changes are being accepted using Cassandra-cli.
>
> I issue the change:
>
> [default@EBonding] update keyspace EBonding
> ... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
> ... and strategy_options={Fisher:2};
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6
> Waiting for schema agreement...
> ... schemas agree across the cluster
>
> Then I do a describe and it still shows the old strategy. Is there something else that
I need to do? I’ve exited and restarted Cassandra-cli and it still shows the SimpleStrategy
for that keyspace. Other nodes show the same information.
>
> [default@EBonding] describe EBonding;
> Keyspace: EBonding:
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> Durable Writes: true
> Options: [replication_factor:2]
>
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Thursday, August 23, 2012 11:06 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> Thanks for the information! Answers my questions.
>
> From: Tyler Hobbs [mailto:tyler@datastax.com]
> Sent: Wednesday, August 22, 2012 7:10 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> We are in the process of building out a new DR system in another Data Center, and we
want to mirror our Cassandra environment to that DR. I have a couple questions on the best
way to do this after reading the documentation on the Datastax website. We didn’t initially
plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we
need to. So I’m just trying to nail down the order of doing this as well as any potential
issues.
>
> For the nodes, we don’t plan on querying the servers in this DR until we fail over
to this data center. We are going to have 5 similar nodes in the DR, should I join them
into the ring at token+1?
>
> Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM
for your queries instead of regular QUORUM.
>
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the
replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with
the updated replication factor for each dr?
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For
the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
>
>
> Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going
to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick
up the value would be ok?
>
> This is the first thing you'll want to do. Unless your node IPs would naturally put
all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using
the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.)
A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties
first if using PFSnitch.
>
>
> This is all on Cassandra 1.1.4, Thanks for any help!
>
>
>
>
>
> --
> Tyler Hobbs
> DataStax
>
>
>
|