incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Jones <MJo...@imagehawk.com>
Subject RE: What is loadbalance supposed to do? 0.6.0RC1
Date Wed, 07 Apr 2010 14:39:46 GMT
The log said Bootstrapping  @ 07:34  (since it was 08:35, I assumed it wasn't doing anything,
also, CPU usage was < 10%)

Turns out, when I restarted the node, it claimed the time was 7:35 rather than 8:35.  Why
would log4j be off by one hour?  We are on CDT here, and have been for more than a week. 
The date command returns the appropriate time (Wed Apr  7 09:24:50 CDT 2010), I see no evidence
of a TZ variable and /etc/timezone shows "America/Chicago"

If it was off by 6 hours instead of 1, I could understand this, but its only off by one hour.

System.getProperties() reports the timezone as blank

Also, if the data is pushed out to the other nodes before the bootstrapping, why has data
been lost?  Does this mean that decommissioning a node results in data loss?



-----Original Message-----
From: Sylvain Lebresne [mailto:sylvain@yakaz.com]
Sent: Wednesday, April 07, 2010 9:07 AM
To: user@cassandra.apache.org
Subject: Re: What is loadbalance supposed to do? 0.6.0RC1

> It shouldn't remove a node from the ring should it?  (appears it did)

It does. As explained here: http://wiki.apache.org/cassandra/Operations,
loadbalance 'decomission' the node and then add it back as a bootstrapping
node (roughly).

So that the node disappear is expected and it is supposed to come back.
But this is not a quick operation (and certainely not one you want to do every
other day). You apparently restarted Cassandra while it was doing its stuff.

Not sure the loss of data is to be expected though.

> It shouldn't remove data from db, should it?  (data size appears to grow, but records
are now missing)
>
> Loaded 38 million "rows" and the ring looked like this:
>
>  mark@ec2:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host 192.168.1.116 ring
>  Address       Status     Load          Range                                      Ring
>                                         167730615856220406399741259265091647472
>  192.168.1.116 Up         4.81 GB       54880762918591020775962843965839761529     |<--|
>  192.168.1.119 Up         12.96 GB      160455137948102479104219052453775170160    |
  |
>  192.168.1.12  Up         8.98 GB       167730615856220406399741259265091647472    |--
>
> So I did this:
>  mark@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host 192.168.1.12
loadbalance
>
> And this happened (even though Cassandra was still running):
>
>  mark@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host 192.168.1.12
ring
>  Address       Status     Load          Range                                      Ring
>                                         160455137948102479104219052453775170160
>  192.168.1.116 Up         12.71 GB      54880762918591020775962843965839761529     |<--|
>  192.168.1.119 Up         13.47 GB      160455137948102479104219052453775170160    |-->|
>
> After restarting Cassandra on .12
>
>  mark@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host 192.168.1.12
ring
>  Address       Status     Load          Range                                      Ring
>                                         160455137948102479104219052453775170160
>  192.168.1.116 Up         12.71 GB      54880762918591020775962843965839761529     |<--|
>  192.168.1.12  Up         8.98 GB       107669873051407416105654071439122680093    |
  |
>  192.168.1.119 Up         13.47 GB      160455137948102479104219052453775170160    |-->|
>
> Now I have more data, but nearly 50% of my queries are failing (not found).  This data
was checked before the load balance was done.
>

Mime
View raw message