incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Hinted handoff
Date Fri, 08 Mar 2013 16:16:13 GMT
> For example in my case running 'upgradesstables' on ~300GB node takes about 30+ hours.
+1
And repair.

>>  1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say 100 nodes…
All of these suggestions on node size boil down to: make sure compaction, repair, node replacement
and node moves complete in a time that is acceptable to you. So would suggest you run some
tests. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 7/03/2013, at 5:06 AM, Michal Michalski <michalm@opera.com> wrote:

> I think it's still true, but not because of network-related issues, but because of the
maintenance problems it will cause during per-node operations. For example in my case running
'upgradesstables' on ~300GB node takes about 30+ hours. The other IO-intensive operations
will probably be a pain too.
> 
> M.
> 
> W dniu 07.03.2013 13:59, Kanwar Sangha pisze:
>> <Do you have 9.5TB on a single node ?>
>> In the normal case it's best to have around 300 to 500GB per node. With that much
data is will take a week to run repair or replace a failed node.
>> 
>> Hi Aaron - This was true for pre 1.2 but with 1.2 and virtual nodes, does this still
hold ? 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say 100 nodes...
>> 
>> 
>> From: aaron morton [mailto:aaron@thelastpickle.com]
>> Sent: 06 March 2013 23:47
>> To: user@cassandra.apache.org
>> Subject: Re: Hinted handoff
>> 
>> Check the IO utilisation using iostat
>> 
>> You *really* should not need to make HH run faster, if you do there is some thing
bad going on. I would consider dropping the hints and running repair.
>> 
>> Data is ~9.5 TB
>> Do you have 9.5TB on a single node ?
>> In the normal case it's best to have around 300 to 500GB per node. With that much
data is will take a week to run repair or replace a failed node.
>> 
>> Cheers
>> 
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 6/03/2013, at 1:22 PM, Kanwar Sangha <kanwar@mavenir.com<mailto:kanwar@mavenir.com>>
wrote:
>> 
>> 
>> Is this correct ?
>> 
>> I have Raid 0 setup for 16 TB across 8 disks. Each disk is 7.2kRPM with IOPS of 80
per disk. Data is ~9.5 TB
>> 
>> So 4K * 80 * 9.5 = 3040 KB ~  23.75 Mb/s.
>> 
>> So basically I am limited at the disk rather than the n/w
>> 
>> From: Kanwar Sangha [mailto:kanwar@mavenir.com<http://mavenir.com>]
>> Sent: 06 March 2013 15:11
>> To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
>> Subject: RE: Hinted handoff
>> 
>> After trying to bump up the "hinted_handoff_throttle_in_kb" to 1G/b per sec, It still
does not go above 25Mb/s.  Is there a limitation ?
>> 
>> 
>> 
>> From: Kanwar Sangha [mailto:kanwar@mavenir.com]
>> Sent: 06 March 2013 14:41
>> To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
>> Subject: RE: Hinted handoff
>> 
>> Got the param. thanks
>> 
>> From: Kanwar Sangha [mailto:kanwar@mavenir.com]
>> Sent: 06 March 2013 13:50
>> To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
>> Subject: Hinted handoff
>> 
>> Hi - Is there a way to increase the hinted handoff throughput ? I am seeing around
8Mb/s (bits).
>> 
>> Thanks,
>> Kanwar
>> 
>> 
> 


Mime
View raw message