Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 93587 invoked from network); 10 Dec 2010 00:17:55 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 10 Dec 2010 00:17:55 -0000 Received: (qmail 88525 invoked by uid 500); 10 Dec 2010 00:17:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88508 invoked by uid 500); 10 Dec 2010 00:17:53 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88500 invoked by uid 99); 10 Dec 2010 00:17:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 00:17:53 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.214.47 is neither permitted nor denied by domain of nick@riptano.com) Received: from [209.85.214.47] (HELO mail-bw0-f47.google.com) (209.85.214.47) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 00:17:48 +0000 Received: by mail-bw0-f47.google.com with SMTP id 10so3376811bwz.34 for ; Thu, 09 Dec 2010 16:17:27 -0800 (PST) Received: by 10.204.99.73 with SMTP id t9mr104323bkn.34.1291940184896; Thu, 09 Dec 2010 16:16:24 -0800 (PST) MIME-Version: 1.0 Received: by 10.204.69.16 with HTTP; Thu, 9 Dec 2010 16:15:37 -0800 (PST) X-Originating-IP: [64.132.24.248] In-Reply-To: References: <4D010F9C.60301@gmail.com> <4D011DDA.30404@code.az> <4D016BB3.7010703@code.az> From: Nick Bailey Date: Thu, 9 Dec 2010 18:15:37 -0600 Message-ID: Subject: Re: Cassandra and disk space To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001485f7bf2ccab7c204970343bb --001485f7bf2ccab7c204970343bb Content-Type: text/plain; charset=ISO-8859-1 Additionally, cleanup will fail to run when the disk is more than 50% full. Another reason to stay below 50%. On Thu, Dec 9, 2010 at 6:03 PM, Tyler Hobbs wrote: > Yes, that's correct, but I wouldn't push it too far. You'll become much > more sensitive to disk usage changes; in particular, rebalancing your > cluster will particularly difficult, and repair will also become dangerous. > Disk performance also tends to drop when a disk nears capacity. > > There's no recommended maximum size -- it all depends on your access > rates. Anywhere from 10 GB to 1TB is typical. > > - Tyler > > > On Thu, Dec 9, 2010 at 5:52 PM, Rustam Aliyev wrote: > >> >> That depends on your scenario. In the worst case of one big CF, there's >> not much that can be easily done for the disk usage of compaction and >> cleanup (which is essentially compaction). >> >> If, instead, you have several column families and no single CF makes up >> the majority of your data, you can push your disk usage a bit higher. >> >> >> Is there any formula to calculate this? Let's say I have 500GB in single >> CF. So I need at least 500GB of free space for compaction. If I partition >> this CF and split it into 10 proportional CFs each 50GB, does it mean that I >> will need only 50GB of free space? >> >> Also, is there recommended maximum of data size per node? >> >> Thanks. >> >> >> A fundamental idea behind Cassandra's architecture is that disk space is >> cheap (which, indeed, it is). If you are particularly sensitive to this, >> Cassandra might not be the best solution to your problem. Also keep in mind >> that Cassandra performs well with average disks, so you don't need to spend >> a lot there. Additionally, most people find that the replication protects >> their data enough to allow them to use RAID 0 instead of 1, 10, 5, or 6. >> >> - Tyler >> >> On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev wrote: >> >>> Is there any plans to improve this in future? >>> >>> For big data clusters this could be very expensive. Based on your >>> comment, I will need 200TB of storage for 100TB of data to keep Cassandra >>> running. >>> >>> -- >>> Rustam. >>> >>> On 09/12/2010 17:56, Tyler Hobbs wrote: >>> >>> If you are on 0.6, repair is particularly dangerous with respect to disk >>> space usage. If your replica is sufficiently out of sync, you can triple >>> your disk usage pretty easily. This has been improved in 0.7, so repairs >>> should use about half as much disk space, on average. >>> >>> In general, yes, keep your nodes under 50% disk usage at all times. Any >>> of: compaction, cleanup, snapshotting, repair, or bootstrapping (the latter >>> two are improved in 0.7) can double your disk usage temporarily. >>> >>> You should plan to add more disk space or add nodes when you get close to >>> this limit. Once you go over 50%, it's more difficult to add nodes, at >>> least in 0.6. >>> >>> - Tyler >>> >>> On Thu, Dec 9, 2010 at 11:19 AM, Mark wrote: >>> >>>> I recently ran into a problem during a repair operation where my nodes >>>> completely ran out of space and my whole cluster was... well, clusterfucked. >>>> >>>> I want to make sure how to prevent this problem in the future. >>>> >>>> Should I make sure that at all times every node is under 50% of its disk >>>> space? Are there any normal day-to-day operations that would cause the any >>>> one node to double in size that I should be aware of? If on or more nodes to >>>> surpass the 50% mark, what should I plan to do? >>>> >>>> Thanks for any advice >>>> >>> >>> >> > --001485f7bf2ccab7c204970343bb Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Additionally, cleanup will fail to run when the disk is more than 50% full.= Another reason to stay below 50%.

On Thu= , Dec 9, 2010 at 6:03 PM, Tyler Hobbs <tyler@riptano.com> wrote:
Yes, that's correct, but I wouldn't= push it too far.=A0 You'll become much more sensitive to disk usage ch= anges; in particular, rebalancing your cluster will particularly difficult,= and repair will also become dangerous.=A0 Disk performance also tends to d= rop when a disk nears capacity.

There's no recommended maximum size -- it all depends on your acces= s rates.=A0 Anywhere from 10 GB to 1TB is typical.

- Tyler


On Thu, Dec 9, 2010 at 5:52 PM, Rustam Aliyev <rustam@code.az> = wrote:
=20 =20 =20

That depends on your scenario.=A0 In the wors= t case of one big CF, there's not much that can be easily done for the disk usage of compaction and cleanup (which is essentially compaction).
If, instead, you have several column families and no single CF makes up the majority of your data, you can push your disk usage a bit higher.


Is there any formula to calculate this? Let's say I have 500GB in single CF. So I need at least 500GB of free space for compaction. If I partition this CF and split it into 10 proportional CFs each 50GB, does it mean that I will need only 50GB of free space?

Also, is there recommended maximum of data size per node?

Thanks.


A fundamental idea behind Cassandra's arc= hitecture is that disk space is cheap (which, indeed, it is).=A0 If you are particularly sensitive to this, Cassandra might not be the best solution to your problem.=A0 Also keep in mind that Cassandra performs well with average disks, so you don't need to spend a lo= t there.=A0 Additionally, most people find that the replication protects their data enough to allow them to use RAID 0 instead of 1, 10, 5, or 6.

- Tyler

On Thu, Dec 9, 2010 at 12:20 PM, Rustam Aliyev <rustam@code.az> wrote:
Is there any plans to improve this in future?

For big data clusters this could be very expensive. Based on your comment, I will need 200TB of storage for 100TB of data to keep Cassandra running.

--
Rustam.

On 09/12/2010 17:56, Tyler Hobbs wrote:
If you are on 0.6, repair is particularly dangerous with respect to disk space usage.=A0 If your replica is sufficiently out of sync, you can triple your disk usage pretty easily.=A0 This has been improved in 0.7, so repairs should use about half as much disk space, on average.

In general, yes, keep your nodes under 50% disk usage at all times.=A0 Any of: compaction, cleanup, snapshotting, repair, or bootstrapping (the latter two are improved in 0.7) can double your disk usage temporarily.

You should plan to add more disk space or add nodes when you get close to this limit.=A0 Once you go over 50%, it's more difficult to add nodes, at least in 0.6.

- Tyler

On Thu, Dec 9, 2010 at 11:19 AM, Mark <static.void.dev@gmail.com> wrote:
I r= ecently ran into a problem during a repair operation where my nodes completely ran out of space and my whole cluster was... well, clusterfucked.

I want to make sure how to prevent this problem in the future.

Should I make sure that at all times every node is under 50% of its disk space? Are there any normal day-to-day operations that would cause the any one node to double in size that I should be aware of? If on or more nodes to surpass the 50% mark, what should I plan to do?

Thanks for any advice




--001485f7bf2ccab7c204970343bb--