Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9F554200BFD for ; Sun, 15 Jan 2017 12:22:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9DC31160B41; Sun, 15 Jan 2017 11:22:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CFCCA160B2B for ; Sun, 15 Jan 2017 12:21:59 +0100 (CET) Received: (qmail 97281 invoked by uid 500); 15 Jan 2017 11:21:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 97271 invoked by uid 99); 15 Jan 2017 11:21:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Jan 2017 11:21:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id F2FF6C09A6 for ; Sun, 15 Jan 2017 11:21:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id hYjjHyo5q8Il for ; Sun, 15 Jan 2017 11:21:53 +0000 (UTC) Received: from mail-ot0-f177.google.com (mail-ot0-f177.google.com [74.125.82.177]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id BBEAD5F254 for ; Sun, 15 Jan 2017 11:21:52 +0000 (UTC) Received: by mail-ot0-f177.google.com with SMTP id 65so29157333otq.2 for ; Sun, 15 Jan 2017 03:21:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=8N9W3AOQd6b5ziTxjwAgJLss0MbvWOtlk7WAAWVuq+A=; b=vf9FlInSvnSafojaH0mdIHG2BWB0/kTDf3RbhQ1Y5hL0doQh4kzNozNO+kPxky+UOu 2BVKL/j1PmXCWHuduh/w343tjZFEwjXLa1xaKSFgikPKAMtNP3WXg038nPvInEGOT9At 7TowZpB7bBf1i0V1uAgiJ+m34hm+cWvJsYfC7IOLWBTBFr0FEm0egeWj3a1YsMArMfN7 +SCiORqPLJQf/gwCJK7zW9sdnDOcqxaddN/lqzct2iLzrwvQDC2AdMVI1UlutslbZmkY 0lQhkCnzrDhiZgwMlvl0wKp+TnbKBJ9v168wncb3WpebRx8t8D0oHxMeTyBKITq/K2v/ wglw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=8N9W3AOQd6b5ziTxjwAgJLss0MbvWOtlk7WAAWVuq+A=; b=pUC3wvfODczAQqcn/8fCcQl9bZYwROF3N7vfBjyFpFx3m59XdDgkwqlbmW7jSf7X9I 904kHS0ntGuK/i8ee5uYE2bPhA0dj0Cnv4hTNa4KCiIQmyqxVB6B9sjDMOs0/OahSDIR 7Fdx30PYQipIyfv7lJHHPdk+hPGgNjjtuLVFkLWrLG5zfKhTRD0hJP/do855e1HF6Yhh AaulHescNLW3dH3q8Oe2Gu/uL2LtBC3PhavuApfzrN05DR4rYGQ82K3uAXrL0H/53ZmP fT3wbrW0YkjdcLLbzYgazQf8rifa+l4QlVnmJPpgaGgev/Z1svni++Lygfrv5d32sYs5 9Zag== X-Gm-Message-State: AIkVDXKk5u+x4icv3P40Dc1zHZnROd3Qdppb6pW4tGsGKKpz1JeZYbE7tzuMKAv1lm5kBlL1JcmDBK46DbM8/A== X-Received: by 10.157.42.202 with SMTP id e68mr14086143otb.221.1484479305786; Sun, 15 Jan 2017 03:21:45 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.53.7 with HTTP; Sun, 15 Jan 2017 03:21:44 -0800 (PST) Received: by 10.157.53.7 with HTTP; Sun, 15 Jan 2017 03:21:44 -0800 (PST) In-Reply-To: References: From: Chris Mawata Date: Sun, 15 Jan 2017 06:21:44 -0500 Message-ID: Subject: Re: Backups eating up disk space To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=94eb2c03750ab2aa4905462045c1 archived-at: Sun, 15 Jan 2017 11:22:01 -0000 --94eb2c03750ab2aa4905462045c1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You don't have a viable solution because you are not making a snapshot as a starting point. After a while you will have a lot of backup data. Using the backups to get your cluster to a given state will involve copying a very large amount of backup data, possibility more than the capacity of your cluster followed by a tremendous amount of compaction. If your topology changes life could really get miserable. I would counsel having period snapshots so that your possible bad day in the future is less bad. On Jan 13, 2017 8:01 AM, "Kunal Gangakhedkar" wrote: > Great, thanks a lot to all for the help :) > > I finally took the dive and went with Razi's suggestions. > In summary, this is what I did: > > - turn off incremental backups on each of the nodes in rolling fashion > - remove the 'backups' directory from each keyspace on each node. > > This ended up freeing up almost 350GB on each node - yay :) > > Again, thanks a lot for the help, guys. > > Kunal > > On 12 January 2017 at 21:15, Khaja, Raziuddin (NIH/NLM/NCBI) [C] < > raziuddin.khaja@nih.gov> wrote: > >> snapshots are slightly different than backups. >> >> >> >> In my explanation of the hardlinks created in the backups folder, notice >> that compacted sstables, never end up in the backups folder. >> >> >> >> On the other hand, a snapshot is meant to represent the data at a >> particular moment in time. Thus, the snapshots directory contains hardli= nks >> to all active sstables at the time the snapshot was taken, which would >> include: compacted sstables; and any sstables from memtable flush or >> streamed from other nodes that both exist in the table directory and the >> backups directory. >> >> >> >> So, that would be the difference between snapshots and backups. >> >> >> >> Best regards, >> >> -Razi >> >> >> >> >> >> *From: *Alain RODRIGUEZ >> *Reply-To: *"user@cassandra.apache.org" >> *Date: *Thursday, January 12, 2017 at 9:16 AM >> >> *To: *"user@cassandra.apache.org" >> *Subject: *Re: Backups eating up disk space >> >> >> >> My 2 cents, >> >> >> >> As I mentioned earlier, we're not currently using snapshots - it's only >> the backups that are bothering me right now. >> >> >> >> I believe backups folder is just the new name for the previously called >> snapshots folder. But I can be completely wrong, I haven't played that m= uch >> with snapshots in new versions yet. >> >> >> >> Anyway, some operations in Apache Cassandra can trigger a snapshot: >> >> >> >> - Repair (when not using parallel option but sequential repairs instead) >> >> - Truncating a table (by default) >> >> - Dropping a table (by default) >> >> - Maybe other I can't think of... ? >> >> >> >> If you want to clean space but still keep a backup you can run: >> >> >> >> "nodetool clearsnapshots" >> >> "nodetool snapshot " >> >> >> >> This way and for a while, data won't be taking space as old files will b= e >> cleaned and new files will be only hardlinks as detailed above. Then you >> might want to work at a proper backup policy, probably implying getting >> data out of production server (a lot of people uses S3 or similar >> services). Or just do that from time to time, meaning you only keep a >> backup and disk space behaviour will be hard to predict. >> >> >> >> C*heers, >> >> ----------------------- >> >> Alain Rodriguez - @arodream - alain@thelastpickle.com >> >> France >> >> >> >> The Last Pickle - Apache Cassandra Consulting >> >> http://www.thelastpickle.com >> >> >> >> 2017-01-12 6:42 GMT+01:00 Prasenjit Sarkar : >> >> Hi Kunal, >> >> >> >> Razi's post does give a very lucid description of how cassandra manages >> the hard links inside the backup directory. >> >> >> >> Where it needs clarification is the following: >> >> --> incremental backups is a system wide setting and so its an all or >> nothing approach >> >> >> >> --> as multiple people have stated, incremental backups do not create >> hard links to compacted sstables. however, this can bloat the size of yo= ur >> backups >> >> >> >> --> again as stated, it is a general industry practice to place backups >> in a different secondary storage location than the main production site.= So >> best to move it to the secondary storage before applying rm on the backu= ps >> folder >> >> >> >> In my experience with production clusters, managing the backups folder >> across multiple nodes can be painful if the objective is to ever recover >> data. With the usual disclaimers, better to rely on third party vendors = to >> accomplish the needful rather than scripts/tablesnap. >> >> >> >> Regards >> >> Prasenjit >> >> >> >> >> >> On Wed, Jan 11, 2017 at 7:49 AM, Khaja, Raziuddin (NIH/NLM/NCBI) [C] < >> raziuddin.khaja@nih.gov> wrote: >> >> Hello Kunal, >> >> >> >> Caveat: I am not a super-expert on Cassandra, but it helps to explain to >> others, in order to eventually become an expert, so if my explanation is >> wrong, I would hope others would correct me. J >> >> >> >> The active sstables/data files are are all the files located in the >> directory for the table. >> >> You can safely remove all files under the backups/ directory and the >> directory itself. >> >> Removing any files that are current hard-links inside backups won=E2=80= =99t cause >> any issues, and I will explain why. >> >> >> >> Have you looked at your Cassandra.yaml file and checked the setting for >> incremental_backups? If it is set to true, and you don=E2=80=99t want t= o make new >> backups, you can set it to false, so that after you clean up, you will n= ot >> have to clean up the backups again. >> >> >> >> Explanation: >> >> Lets look at the the definition of incremental backups again: =E2=80=9CC= assandra >> creates a hard link to each SSTable flushed or streamed locally in >> a backups subdirectory of the keyspace data.=E2=80=9D >> >> >> >> Suppose we have a directory path: my_keyspace/my_table-some-uuid/backups= / >> >> In the rest of the discussion, when I refer to =E2=80=9Ctable directory= =E2=80=9D, I >> explicitly mean the directory: my_keyspace/my_table-some-uuid/ >> >> When I refer to backups/ directory, I explicitly mean: >> my_keyspace/my_table-some-uuid/backups/ >> >> >> >> Suppose that you have an sstable-A that was either flushed from a >> memtable or streamed from another node. >> >> At this point, you have a hardlink to sstable-A in your table directory, >> and a hardlink to sstable-A in your backups/ directory. >> >> Suppose that you have another sstable-B that was also either flushed fro= m >> a memtable or streamed from another node. >> >> At this point, you have a hardlink to sstable-B in your table directory, >> and a hardlink to sstable-B in your backups/ directory. >> >> >> >> Next, suppose compaction were to occur, where say sstable-A and sstable-= B >> would be compacted to produce sstable-C, representing all the data from = A >> and B. >> >> Now, sstable-C will live in your main table directory, and the hardlinks >> to sstable-A and sstable-B will be deleted in the main table directory, = but >> sstable-A and sstable-B will continue to exist in /backups. >> >> At this point, in your main table directory, you will have a hardlink to >> sstable-C. In your backups/ directory you will have hardlinks to sstable= -A, >> and sstable-B. >> >> >> >> Thus, your main table directory is not cluttered with old un-compacted >> sstables, and only has the sstables along with other files that are >> actively being used. >> >> >> >> To drive the point home, =E2=80=A6 >> >> Suppose that you have another sstable-D that was either flushed from a >> memtable or streamed from another node. >> >> At this point, in your main table directory, you will have sstable-C and >> sstable-D. In your backups/ directory you will have hardlinks to sstable= -A, >> sstable-B, and sstable-D. >> >> >> >> Next, suppose compaction were to occur where say sstable-C and sstable-D >> would be compacted to produce sstable-E, representing all the data from = C >> and D. >> >> Now, sstable-E will live in your main table directory, and the hardlinks >> to sstable-C and sstable-D will be deleted in the main table directory, = but >> sstable-D will continue to exist in /backups. >> >> At this point, in your main table directory, you will have a hardlink to >> sstable-E. In your backups/ directory you will have hardlinks to sstable= -A, >> sstable-B and sstable-D. >> >> >> >> As you can see, the /backups directory quickly accumulates with all >> un-compacted sstables and how it progressively used up more and more spa= ce. >> >> Also, note that the /backups directory does not contain sstables >> generated from compaction, such as sstable-C and sstable-E. >> >> It is safe to delete the entire backups/ directory because all the data >> is represented in the compacted sstable-E. >> >> I hope this explanation was clear and gives you confidence in using rm t= o >> delete the directory for backups/. >> >> >> >> Best regards, >> >> -Razi >> >> >> >> >> >> >> >> *From: *Kunal Gangakhedkar >> *Reply-To: *"user@cassandra.apache.org" >> *Date: *Wednesday, January 11, 2017 at 6:47 AM >> >> >> *To: *"user@cassandra.apache.org" >> *Subject: *Re: Backups eating up disk space >> >> >> >> Thanks for the reply, Razi. >> >> >> >> As I mentioned earlier, we're not currently using snapshots - it's only >> the backups that are bothering me right now. >> >> >> >> So my next question is pertaining to this statement of yours: >> >> >> >> As far as I am aware, using *rm* is perfectly safe to delete the >> directories for snapshots/backups as long as you are careful not to dele= te >> your actively used sstable files and directories. >> >> >> >> How do I find out which are the actively used sstables? >> >> If by that you mean the main data files, does that mean I can safely >> remove all files ONLY under the "backups/" directory? >> >> Or, removing any files that are current hard-links inside backups can >> potentially cause any issues? >> >> >> >> Thanks, >> >> Kunal >> >> >> >> On 11 January 2017 at 01:06, Khaja, Raziuddin (NIH/NLM/NCBI) [C] < >> raziuddin.khaja@nih.gov> wrote: >> >> Hello Kunal, >> >> >> >> I would take a look at the following configuration options in the >> Cassandra.yaml >> >> >> >> *Common automatic backup settings* >> >> *Incremental_backups:* >> >> http://docs.datastax.com/en/archived/cassandra/3.x/cassandra >> /configuration/configCassandra_yaml.html#configCassandra_ >> yaml__incremental_backups >> >> >> >> (Default: false) Backs up data updated since the last snapshot was taken= . >> When enabled, Cassandra creates a hard link to each SSTable flushed or >> streamed locally in a backups subdirectory of the keyspace data. Removin= g >> these links is the operator's responsibility. >> >> >> >> *snapshot_before_compaction*: >> >> http://docs.datastax.com/en/archived/cassandra/3.x/cassandra >> /configuration/configCassandra_yaml.html#configCassandra_ >> yaml__snapshot_before_compaction >> >> >> >> (Default: false) Enables or disables taking a snapshot before each >> compaction. A snapshot is useful to back up data when there is a data >> format change. Be careful using this option: Cassandra does not clean up >> older snapshots automatically. >> >> >> >> >> >> *Advanced automatic backup setting* >> >> *auto_snapshot*: >> >> http://docs.datastax.com/en/archived/cassandra/3.x/cassandra >> /configuration/configCassandra_yaml.html#configCassandra_ >> yaml__auto_snapshot >> >> >> >> (Default: true) Enables or disables whether Cassandra takes a snapshot o= f >> the data before truncating a keyspace or dropping a table. To prevent da= ta >> loss, Datastax strongly advises using the default setting. If you >> set auto_snapshot to false, you lose data on truncation or drop. >> >> >> >> >> >> *nodetool* also provides methods to manage snapshots. >> http://docs.datastax.com/en/archived/cassandra/3.x/cassandra >> /tools/toolsNodetool.html >> >> See the specific commands: >> >> - nodetool clearsnapshot >> >> Removes one or more snapshots. >> - nodetool listsnapshots >> >> Lists snapshot names, size on disk, and true size. >> - nodetool snapshot >> >> Take a snapshot of one or more keyspaces, or of a table, to backup >> data. >> >> >> >> As far as I am aware, using *rm* is perfectly safe to delete the >> directories for snapshots/backups as long as you are careful not to dele= te >> your actively used sstable files and directories. I think the *nodetool >> clearsnapshot* command is provided so that you don=E2=80=99t accidentall= y delete >> actively used files. Last I used *clearsnapshot*, (a very long time >> ago), I thought it left behind the directory, but this could have been >> fixed in newer versions (so you might want to check that). >> >> >> >> HTH >> >> -Razi >> >> >> >> >> >> *From: *Jonathan Haddad >> *Reply-To: *"user@cassandra.apache.org" >> *Date: *Tuesday, January 10, 2017 at 12:26 PM >> *To: *"user@cassandra.apache.org" >> *Subject: *Re: Backups eating up disk space >> >> >> >> If you remove the files from the backup directory, you would not have >> data loss in the case of a node going down. They're hard links to the s= ame >> files that are in your data directory, and are created when an sstable i= s >> written to disk. At the time, they take up (almost) no space, so they >> aren't a big deal, but when the sstable gets compacted, they stick aroun= d, >> so they end up not freeing space up. >> >> >> >> Usually you use incremental backups as a means of moving the sstables of= f >> the node to a backup location. If you're not doing anything with them, >> they're just wasting space and you should disable incremental backups. >> >> >> >> Some people take snapshots then rely on incremental backups. Others use >> the tablesnap utility which does sort of the same thing. >> >> >> >> On Tue, Jan 10, 2017 at 9:18 AM Kunal Gangakhedkar < >> kgangakhedkar@gmail.com> wrote: >> >> Thanks for quick reply, Jon. >> >> >> >> But, what about in case of node/cluster going down? Would there be data >> loss if I remove these files manually? >> >> >> >> How is it typically managed in production setups? >> >> What are the best-practices for the same? >> >> Do people take snapshots on each node before removing the backups? >> >> >> >> This is my first production deployment - so, still trying to learn. >> >> >> >> Thanks, >> >> Kunal >> >> >> >> On 10 January 2017 at 21:36, Jonathan Haddad wrote: >> >> You can just delete them off the filesystem (rm) >> >> >> >> On Tue, Jan 10, 2017 at 8:02 AM Kunal Gangakhedkar < >> kgangakhedkar@gmail.com> wrote: >> >> Hi all, >> >> >> >> We have a 3-node cassandra cluster with incremental backup set to true. >> >> Each node has 1TB data volume that stores cassandra data. >> >> >> >> The load in the output of 'nodetool status' comes up at around 260GB eac= h >> node. >> >> All our keyspaces use replication factor =3D 3. >> >> >> >> However, the df output shows the data volumes consuming around 850GB of >> space. >> >> I checked the keyspace directory structures - most of the space goes in >> /data///backups. >> >> >> >> We have never manually run snapshots. >> >> >> >> What is the typical procedure to clear the backups? >> >> Can it be done without taking the node offline? >> >> >> >> Thanks, >> >> Kunal >> >> >> >> >> >> >> >> >> > > --94eb2c03750ab2aa4905462045c1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

You don't have a viable solution because you are not mak= ing a snapshot as a starting point. After a while you will have a lot of ba= ckup data.=C2=A0 Using the backups to get your cluster to a given state wil= l involve copying a very large amount of backup data, possibility more than= the capacity of your cluster followed by a tremendous amount of compaction= . If your topology changes life could really get miserable. I would counsel= having period snapshots so that your possible bad day in the future is les= s bad.

On Jan 13, 2017 8:01 AM, "Kunal Gangakhedka= r" <kgangakhedkar@gmail.= com> wrote:
<= div dir=3D"ltr">Great, thanks a lot to all for the help :)

I finally took the dive and went with Razi's suggestions.
= In summary, this is what I did:
  • turn off incremental back= ups on each of the nodes in rolling fashion
  • remove the 'bac= kups' directory from each keyspace on each node.
This= ended up freeing up almost 350GB on each node - yay :)
Again, thanks a lot for the help, guys.

Kunal

On 12 January 2017 at 21:15, Khaja, Raziuddi= n (NIH/NLM/NCBI) [C] <raziuddin.khaja@nih.gov> wrote:<= br>

snapshots are slightly different than backups.

=C2=A0

In my explanation of the hardlinks created in the backups folder, notice t= hat compacted sstables, never end up in the backups folder.

=C2=A0

On the other hand, a snapshot is meant to represent the data at a particul= ar moment in time. Thus, the snapshots directory contains hardlinks to all = active sstables at the time the snapshot was taken, which would include: compacted sstables; and any sstables from = memtable flush or streamed from other nodes that both exist in the table di= rectory and the backups directory.

=C2=A0

So, that would be the difference between snapshots and backups.<= /u>

=C2=A0

Best regards,

-Razi

=C2=A0

=C2=A0

F= rom: Alain RODRIGUEZ <arodrime@gmail.com= >
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Thursday, January 12, 2017 at 9:16 AM


To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Backups eating up disk space
<= p>

=C2=A0

My 2 cents,

=C2=A0

As I mentioned earli= er, we're not currently using snapshots - it's only the backups tha= t are bothering me right now.

=C2=A0

I believe backups folder is just the new name for th= e previously called snapshots folder. But I can be completely wrong, I have= n't played that much with snapshots in new versions yet.<= /p>

=C2=A0

Anyway, some operations in Apache Cassandra can trig= ger a snapshot:

=C2=A0

- Repair (when not using parallel option but sequent= ial repairs instead)

- Truncating a table (by default)

- Dropping a table (by default)

- Maybe other I can't think of... ?

=C2=A0

If you want to clean space but still keep a backup y= ou can run:

=C2=A0

"nodetool clearsnapshots"

"nodetool snapshot <whatever>"

=C2=A0

This way and for a while, data won't be taking s= pace as old files will be cleaned and new files will be only hardlinks as d= etailed above. Then you might want to work at a proper backup policy, proba= bly implying getting data out of production server (a lot of people uses S3 or similar services). Or just do that from= time to time, meaning you only keep a backup and disk space behaviour will= be hard to predict.

=C2=A0

C*heers,

-----------------------

Alain Rodriguez - @arodream - alain@thelastpickle.com

France

=C2=A0

The Last Pickle - Apache Cassandra Consulting=

=C2=A0

2017-01-12 6:42 GMT+01:00 Prasenjit Sarkar <prasenjit.sarkar@= datos.io>:

Hi Kunal,

=C2=A0

Razi's post does give a very lucid description o= f how cassandra manages the hard links inside the backup directory.<= u>

=C2=A0

Where it needs clarification is the following:

--> incremental backups is a system wide setting = and so its an all or nothing approach

=C2=A0

--> as multiple people have stated, incremental b= ackups do not create hard links to compacted sstables. however, this can bl= oat the size of your backups

=C2=A0

--> again as stated, it is a general industry pra= ctice to place backups in a different secondary storage location than the m= ain production site. So best to move it to the secondary storage before app= lying rm on the backups folder

=C2=A0

In my experience with production clusters, managing = the backups folder across multiple nodes can be painful if the objective is= to ever recover data. With the usual disclaimers, better to rely on third = party vendors to accomplish the needful rather than scripts/tablesnap.

=C2=A0

Regards

Prasenjit

=C2=A0

=C2=A0

On Wed, Jan 11, 2017 at 7:49 AM, Khaja, Raziuddin (N= IH/NLM/NCBI) [C] <raziuddin.khaja@nih.gov> wrote:

Hello Kunal,

=C2=A0

Caveat: I am not a super-expert on Cassandra, but it helps to explain to o= thers, in order to eventually become an expert, so if my explanation is wrong, I would hope others would correct me. J

=C2=A0

The active sstables/data files are are all the files located in the direct= ory for the table.

You can safely remove all files under the backups/ directory and the direc= tory itself.

Removing any files that are current hard-links inside backups won=E2=80=99= t cause any issues, and I will explain why.

=C2=A0

Have you looked at your Cassandra.yaml file and checked the setting for in= cremental_backups?=C2=A0 If it is set to true, and you don=E2=80=99t want to make new backups, you can set it to false, so that after you clean= up, you will not have to clean up the backups again.<= /p>

=C2=A0

Explanation:

Lets look at the the definition of incremental backups again: =E2=80=9CCas= sandra creates a hard link to each SSTable flushed or streamed locally in a=C2=A0backups=C2=A0subdirectory of the keyspace data.=E2=80=9D=

=C2=A0

Suppose we have a directory path: my_keyspace/my_table-some-uuid/back= ups/

In the rest of the discussion, when I refer to =E2=80=9Ctable directory=E2= =80=9D, I explicitly mean the directory: my_keyspace/my_table-some-uuid/

When I refer to backups/ directory, I explicitly mean: my_keyspace/my_tabl= e-some-uuid/backups/

=C2=A0

Suppose that you have an sstable-A that was either flushed from a memtable= or streamed from another node.

At this point, you have a hardlink to sstable-A in your table directory, a= nd a hardlink to sstable-A in your backups/ directory.=

Suppose that you have another sstable-B that was also either flushed from = a memtable or streamed from another node.

At this point, you have a hardlink to sstable-B in your table directory, a= nd a hardlink to sstable-B in your backups/ directory.=

=C2=A0

Next, suppose compaction were to occur, where say sstable-A and sstable-B = would be compacted to produce sstable-C, representing all the data from A and B.

Now, sstable-C will live in your main table directory, and the hardlinks t= o sstable-A and sstable-B will be deleted in the main table directory, but sstable-A and sstable-B will continue to exist in /ba= ckups.

At this point, in your main table directory, you will have a hardlink to s= stable-C. In your backups/ directory you will have hardlinks to sstable-A, and sstable-B.

=C2=A0

Thus, your main table directory is not cluttered with old un-compacted sst= ables, and only has the sstables along with other files that are actively being used.

=C2=A0

To drive the point home, =E2=80=A6

Suppose that you have another sstable-D that was either flushed from a mem= table or streamed from another node.

At this point, in your main table directory, you will have sstable-C and s= stable-D. In your backups/ directory you will have hardlinks to sstable-A, sstable-B, and sstable-D.

=C2=A0

Next, suppose compaction were to occur where say sstable-C and sstable-D w= ould be compacted to produce sstable-E, representing all the data from C and D.

Now, sstable-E will live in your main table directory, and the hardlinks t= o sstable-C and sstable-D will be deleted in the main table directory, but sstable-D will continue to exist in /backups. =

At this point, in your main table directory, you will have a hardlink to s= stable-E. In your backups/ directory you will have hardlinks to sstable-A, sstable-B and sstable-D.

=C2=A0

As you can see, the /backups directory quickly accumulates with all un-com= pacted sstables and how it progressively used up more and more space.

Also, note that the /backups directory does not contain sstables generated= from compaction, such as sstable-C and sstable-E.

It is safe to delete the entire backups/ directory because all the data is= represented in the compacted sstable-E.

I hope this explanation was clear and gives you confidence in using rm to = delete the directory for backups/.

=C2=A0

Best regards,

-Razi

=C2=A0

=C2=A0

=C2=A0

F= rom: Kunal Gangakhedk= ar <kgangak= hedkar@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Wednesday, January 11, 2017 at 6:47 AM


To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Backups eating up disk space

=C2=A0

Thanks for the reply, Razi.

=C2=A0

As I mentioned earlier, we're not currently usin= g snapshots - it's only the backups that are bothering me right now.=

=C2=A0

So my next question is pertaining to this statement = of yours:

=C2=A0

As far as I am aware, using=C2=A0rm=C2=A0is perfectly safe to delet= e the directories for snapshots/backups as long as you are careful not to delete your actively used sstable files and directories.=C2=A0

=C2=A0

How do I find out which are the actively used sstabl= es?

If by that you mean the main data files, does that m= ean I can safely remove all files ONLY under the "backups/" direc= tory?

Or, removing any files that are current hard-links i= nside backups can potentially cause any issues?

=C2=A0

Thanks,

Kunal

=C2=A0

On 11 January 2017 at 01:06, Khaja, Raziuddin (NIH/N= LM/NCBI) [C] <raziuddin.khaja@nih.gov> wrote:

Hello Kunal,

=C2=A0

I would take a look at the following configuration options in the Cassandr= a.yaml

=C2=A0

Common automatic backup settings

Incremental_backups:

http://docs.datastax.com/en/archived/cassandra= /3.x/cassandra/configuration/configCassandra_yaml.html#configCass= andra_yaml__incremental_backups

=C2=A0

(Default:=C2=A0false) = Backs up data updated since the last snapshot was taken. When enabled, Cass= andra creates a hard link to each SSTable flushed or streamed locally in a= =C2=A0backups=C2=A0subdirectory of the keyspace data. Removing these links is the operator's responsibility.

=C2=A0

snapshot_before_compaction:

http://docs.datastax.com/en/archived/ca= ssandra/3.x/cassandra/configuration/configCassandra_yaml.html#con= figCassandra_yaml__snapshot_before_compaction

=C2=A0

(Default:=C2=A0false) = Enables or disables taking a snapshot before each compaction. A snapshot is= useful to back up data when there is a data format change. Be careful usin= g this option: Cassandra does not clean up older snapshots automatically.

=C2=A0

=C2=A0

Advanced automatic backup setting

auto_snapshot:

http://docs.datastax.com/en/archived/cassandra/3.x/c= assandra/configuration/configCassandra_yaml.html#configCassandra_= yaml__auto_snapshot

=C2=A0

(Default:=C2=A0true) E= nables or disables whether Cassandra takes a snapshot of the data before tr= uncating a keyspace or dropping a table. To prevent data loss, Datastax str= ongly advises using the default setting. If you set=C2=A0auto_snapshot=C2=A0to=C2=A0false, you lose data on truncat= ion or drop.

=C2=A0

=C2=A0

nodetool also provides methods to manage snapshots. http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/tool= s/toolsNodetool.html

See the specific commands:

=C2=A0

As far as I am aware, using rm is perfectly safe to delete the directories for snapshots/backups= as long as you are careful not to delete your actively used sstable files = and directories.=C2=A0 I think the nodetool clearsnapshot command is provided so that you don=E2=80=99t= accidentally delete actively used files.=C2=A0 Last I used clearsnapshot, (a very long time ago), I thought it left behind the = directory, but this could have been fixed in newer versions (so you might w= ant to check that).

=C2=A0

HTH

-Razi

=C2=A0

=C2=A0

F= rom: Jonathan Haddad = <jon@jonhaddad.co= m>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Tuesday, January 10, 2017 at 12:26 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Backups eating up disk space

=C2=A0

If you remove the files from the backup directory, y= ou would not have data loss in the case of a node going down.=C2=A0 They= 9;re hard links to the same files that are in your data directory, and are created when an sstable is written to disk.=C2=A0 At the time, the= y take up (almost) no space, so they aren't a big deal, but when the ss= table gets compacted, they stick around, so they end up not freeing space u= p. =C2=A0

=C2=A0

Usually you use incremental backups as a means of mo= ving the sstables off the node to a backup location.=C2=A0 If you're no= t doing anything with them, they're just wasting space and you should disable incremental backups.

=C2=A0

Some people take snapshots then rely on incremental = backups.=C2=A0 Others use the tablesnap utility which does sort of the same= thing.

=C2=A0

On Tue, Jan 10, 2017 at 9:18 AM Kunal Gangakhedkar &= lt;kgangakhedk= ar@gmail.com> wrote:

Thanks for quick reply, Jon.

=C2=A0

But, what about in case of node/cluster going down? = Would there be data loss if I remove these files manually?

=C2=A0

How is it typically managed in production setups?=

What are the best-practices for the same?<= /u>

Do people take snapshots on each node before removin= g the backups?

=C2=A0

This is my first production deployment - so, still t= rying to learn.

=C2=A0

Thanks,

Kunal

=C2=A0

On 10 January 2017 at 21:36, Jonathan Haddad <jon@jonhaddad.co= m> wrote:

You can just delete them off the filesystem (rm)<= /u>

=C2=A0

On Tue, Jan 10, 2017 at 8:02 AM Kunal Gangakhedkar &= lt;kgangakhedk= ar@gmail.com> wrote:

Hi all,

=C2=A0

We have a 3-node cassandra cluster with incremental = backup set to true.

Each node has 1TB data volume that stores cassandra = data.

=C2=A0

The load in the output of 'nodetool status' = comes up at around 260GB each node.

All our keyspaces use replication factor =3D 3.

=C2=A0

However, the df output shows the data volumes consum= ing around 850GB of space.

I checked the keyspace directory structures - most o= f the space goes in <CASS_DATA_VOL>/data/<KEYSPACE>/<CF= >/backups.

=C2=A0

We have never manually run snapshots.<= /p>

=C2=A0

What is the typical procedure to clear the backups?<= u>

Can it be done without taking the node offline?

=C2=A0

Thanks,

Kunal

=C2=A0

=C2=A0

=C2=A0

=C2=A0


--94eb2c03750ab2aa4905462045c1--