incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Fraenkel <ta...@tok-media.com>
Subject Re: running two rings on the same subnet
Date Tue, 06 Mar 2012 07:12:53 GMT
Works..

But during the night my setup encountered a problem.
I have two VMs on my cluster (running on VmWare ESXi).
Each VM has1GB memory, and two Virtual Disks of 16 GB
They are running on a small server with 4CPUs (2.66 GHz), and 4 GB memory
(together with two other VMs)
I put cassandra data on the second disk of each machine.
VMs are running Ubuntu 11.10 and cassandra 1.0.7.

I left them running overnight and this morning when I came:
In one node cassandra was down, and the last thing in the system.log is:

 INFO [CompactionExecutor:150] 2012-03-06 00:55:04,821 CompactionTask.java
(line 113) Compacting
[SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1243-Data.db'),
SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1245-Data.db'),
SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1242-Data.db'),
SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db')]
 INFO [CompactionExecutor:150] 2012-03-06 00:55:07,919 CompactionTask.java
(line 221) Compacted to
[/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1246-Data.db,].
 32,424,771 to 26,447,685 (~81% of original) bytes for 58,938 keys at
8.144165MB/s.  Time: 3,097ms.


The other node was using all it's CPU and I had to restart it.
After that, I can see that the last lines in it's system.log are that the
other node is down...

 INFO [FlushWriter:142] 2012-03-06 00:55:02,418 Memtable.java (line 246)
Writing Memtable-tk_vertical_tag_story_indx@1365852701(1122169/25154556
serialized/live bytes, 21173 ops)
 INFO [FlushWriter:142] 2012-03-06 00:55:02,742 Memtable.java (line 283)
Completed flushing
/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db (2075930
bytes)
 INFO [GossipTasks:1] 2012-03-06 08:02:18,584 Gossiper.java (line 818)
InetAddress /10.0.0.31 is now dead.

How can I trace why that happened?
Also, I brought cassandra up in both nodes. They both spend long time
reading commit logs, but now they seem to run.
Any idea how to debug or improve my setup?
Thanks,
Tamar



*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Mon, Mar 5, 2012 at 7:30 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Create nodes that do not share seeds, and give the clusters different
> names as a safety measure.
>
> Cheers
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6/03/2012, at 12:04 AM, Tamar Fraenkel wrote:
>
> I want tow separate clusters.
> *Tamar Fraenkel *
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
>
> tamar@tok-media.com
> Tel:   +972 2 6409736
> Mob:  +972 54 8356490
> Fax:   +972 2 5612956
>
>
>
>
>
> On Mon, Mar 5, 2012 at 12:48 PM, aaron morton <aaron@thelastpickle.com>wrote:
>
>> Do you want to create two separate clusters or a single cluster with two
>> data centres ?
>>
>> If it's the later, token selection is discussed here
>> http://www.datastax.com/docs/1.0/install/cluster_init#token-gen-cassandra
>>
>>
>> Moreover all tokens must be unique (even across datacenters), although -
>> from pure curiosity - I wonder what is the rationale behind this.
>>
>> Otherwise data is not evenly distributed.
>>
>> By the way, can someone enlighten me about the first line in the output
>> of the nodetool. Obviously it contains a token, but nothing else. It seems
>> like a formatting glitch, but maybe it has a role.
>>
>> It's the exclusive lower bound token for the first node in the ring. This
>> also happens to be the token for the last node in the ring.
>>
>> In your setup
>> 10.0.0.19 "owns" (85070591730234615865843651857942052864+1) to 0
>> 10.0.0.28 "owns"  (0 + 1) to 85070591730234615865843651857942052864
>>
>> (does not imply primary replica, just used to map keys to nodes.)
>>
>>
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 5/03/2012, at 11:38 PM, Hontvári József Levente wrote:
>>
>>  You have to use PropertyFileSnitch and NetworkTopologyStrategy to create
>> a multi-datacenter setup with two circles. You can start reading from this
>> page:
>>
>> http://www.datastax.com/docs/1.0/cluster_architecture/replication#about-replica-placement-strategy
>>
>> Moreover all tokens must be unique (even across datacenters), although -
>> from pure curiosity - I wonder what is the rationale behind this.
>>
>> By the way, can someone enlighten me about the first line in the output
>> of the nodetool. Obviously it contains a token, but nothing else. It seems
>> like a formatting glitch, but maybe it has a role.
>>
>> On 2012.03.05. 11:06, Tamar Fraenkel wrote:
>>
>> Hi!
>> I have a Cassandra  cluster with two nodes
>>
>>  nodetool ring -h localhost
>> Address         DC          Rack        Status State   Load
>>  Owns    Token
>>
>>      85070591730234615865843651857942052864
>> 10.0.0.19       datacenter1 rack1       Up     Normal  488.74 KB
>> 50.00%  0
>> 10.0.0.28       datacenter1 rack1       Up     Normal  504.63 KB
>> 50.00%  85070591730234615865843651857942052864
>>
>>  I want to create a second ring with the same name but two different
>> nodes.
>> using tokengentool I get the same tokens as they are affected from the
>> number of nodes in a ring.
>>
>>  My question is like this:
>> Lets say I create two new VMs, with IPs: 10.0.0.31 and 10.0.0.11
>> *In 10.0.0.31 cassandra.yaml I will set*
>> initial_token: 0
>> seeds: "10.0.0.31"
>> listen_address: 10.0.0.31
>> rpc_address: 0.0.0.0
>>
>>  *In 10.0.0.11 cassandra.yaml I will set*
>> initial_token: 85070591730234615865843651857942052864
>> seeds: "10.0.0.31"
>> listen_address: 10.0.0.11
>> rpc_address: 0.0.0.0
>>
>>  *Would the rings be separate?*
>>
>>  Thanks,
>>
>>  *Tamar Fraenkel *
>> Senior Software Engineer, TOK Media
>>
>> <Mail Attachment.png>
>>
>>
>> tamar@tok-media.com
>> Tel:   +972 2 6409736
>> Mob:  +972 54 8356490
>> Fax:   +972 2 5612956
>>
>>
>>
>>
>>
>>
>
>

Mime
View raw message