incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Dworkis <...@mylife.com>
Subject Re: curious space usages after recovering a failed node
Date Wed, 18 Aug 2010 18:25:00 GMT
to update, i seem to be having luck with some combination of "cleanup" 
followed by triggering a garbage collection on jmx (all on each node).

(using jxterm):

echo -e 'open localhost:8080\nrun -b java.lang:type=Memory gc' | java -jar jmxterm-1.0-alpha-4-uber.jar

-scott

On Mon, 16 Aug 2010, Scott Dworkis wrote:

> i followed the alternative approach for handling a failed node here:
>
> http://wiki.apache.org/cassandra/Operations
>
> i.e. bringing up a replacement node with the same ip, bootstrapping it into 
> the same token used by the failed node (using the InitialToken config 
> parameter), then doing a repair.  at the end of this process i had a data 
> directory that was almost 3x the size of the directory on the failed node at 
> the time of failure... i expected around 2x for copies going around, but 3x 
> seems a bit high for headroom i should expect to need for recovery.
>
> the data i inserted here was 100 copies of an almost 10M file, random 
> partitioner, no overwriting or anything, replication factor of 2.  so i'd 
> expect to be using around 2G.
>
> here is what ring and du looked like after the initial data load:
>
> Address       Status     Load          Range 
> Ring
>                                       170141183460469231731687303715884105728 
> 10.3.0.84     Up         448.8 MB      42535295865117307932921825928971026432 
> |<--|
> 10.3.0.85     Up         374 MB        85070591730234615865843651857942052864 
> |   |
> 10.3.0.114    Up         495 bytes 
> 127605887595351923798765477786913079296    |   |
> 10.3.0.115    Up         496 bytes 
> 170141183460469231731687303715884105728    |-->|
>
> 655M    /data/cassandra/
> 655M    /data/cassandra
> 655M    /data/cassandra
> 1001M     /data/cassandra
>
>
> so far so good... now after the bootstrap:
>
> Address       Status     Load          Range 
> Ring
>                                       170141183460469231731687303715884105728 
> 10.3.0.84     Up         467.5 MB      42535295865117307932921825928971026432 
> |<--|
> 10.3.0.85     Up         205.7 MB      85070591730234615865843651857942052864 
> |   |
> 10.3.0.114    Up         448.8 MB 
> 127605887595351923798765477786913079296    |   |
> 10.3.0.115    Up         514.25 MB 
> 170141183460469231731687303715884105728    |-->|
>
> 674M    /data/cassandra
> 206M    /data/cassandra/
> 655M    /data/cassandra
> 767M      /data/cassandra
>
>
> also reasonable, now after the repair:
>
> Address       Status     Load          Range 
> Ring
>                                       170141183460469231731687303715884105728 
> 10.3.0.84     Up         467.5 MB      42535295865117307932921825928971026432 
> |<--|
> 10.3.0.85     Up         916.3 MB      85070591730234615865843651857942052864 
> |   |
> 10.3.0.114    Up         654.5 MB 
> 127605887595351923798765477786913079296    |   |
> 10.3.0.115    Up         514.25 MB 
> 170141183460469231731687303715884105728    |-->|
>
> 674M    /data/cassandra
> 1.4G    /data/cassandra/
> 655M    /data/cassandra
> 767M      /data/cassandra
>
>
> so i need 3x headroom if i were to try this on a huge production data set? 
> after 3 or 4 nodetool cleanups, ring looks ok, but the data directories have 
> bloated:
>
> Address       Status     Load          Range 
> Ring
>                                       170141183460469231731687303715884105728 
> 10.3.0.84     Up         467.5 MB      42535295865117307932921825928971026432 
> |<--|
> 10.3.0.85     Up         420.75 MB     85070591730234615865843651857942052864 
> |   |
> 10.3.0.114    Up         448.8 MB 
> 127605887595351923798765477786913079296    |   |
> 10.3.0.115    Up         514.25 MB 
> 170141183460469231731687303715884105728    |-->|
>
> 1.2G    /data/cassandra
> 842M    /data/cassandra/
> 1.1G    /data/cassandra
> 1.3G      /data/cassandra
>
>
> so the question is, do i plan to need 3x headroom for node recoveries?
>
> -scott
>

Mime
View raw message