Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 00327D6EC for ; Thu, 18 Oct 2012 11:32:37 +0000 (UTC) Received: (qmail 72452 invoked by uid 500); 18 Oct 2012 11:32:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 72128 invoked by uid 500); 18 Oct 2012 11:32:32 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72077 invoked by uid 99); 18 Oct 2012 11:32:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Oct 2012 11:32:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a59.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Oct 2012 11:32:24 +0000 Received: from homiemail-a59.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTP id 32D8556406C for ; Thu, 18 Oct 2012 04:32:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=0rL13SUKJYCHCoqieu3AYCDfQ9 s=; b=KXyFlk2E1HEiFflVqg9OOm4m5QYzlKCvU6yiot9NP7KXMCHvA2uc+KjpNx aTvkCrcSiZNdlHOV4Ioe4e+vvOcNgWzizqP2ZfbVmtBi2DKAexg3zMt+uCYL9WVX RFuLbKOv+dBvuCNv0JUdvF4keWaPvdzma/Ml2m3NQDWqzwpMA= Received: from [172.16.1.10] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTPSA id 602E956405C for ; Thu, 18 Oct 2012 04:32:05 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_308CA26B-7F71-4192-BC3B-12D6941739A7" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Subject: Re: RF update Date: Fri, 19 Oct 2012 00:32:02 +1300 References: <868AB5E4-E058-4A16-BDDB-A5849A0155E9@voodoolunchbox.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1498) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_308CA26B-7F71-4192-BC3B-12D6941739A7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > Follow up question: Is it safe to abort the compactions happening = after node repair? It is always safe to abort a compaction. The purpose of compaction is to = replicate the current truth in a more compact format. It does not modify = data, it just creates new files. The worse case would be killing it = between the time the new files are marked as non temp and the time the = old files are deleted. That would result in wasted disk space, but the = truth in the system would not change.=20 >=20 > > Question: These additional compactions seem redundant since there = are no reads or writes on the cluster after the first major compaction = (immediately after the data load), is that right? Repair transfers a portion of the -Data.db component from potentially = multiple SSTables. This may result in multiple new SStables being = created on the receiving node. Once the files are created they are = processed in a similar way to when a memtable is flushed and so = compaction kicks in. > And if so, what can we do to avoid them? We are currently waiting = multiple days. That fact that compaction is taking so long is odd. Have you checked the = logs for GC problems? if you are running an SSD backed instance and have = turned off compaction throttling the high IO throughput can result in = mucho garbage. Faster is not always better.=20 To improve your situation consider: * disabling compaction by setting min_compaction_threshold and = max_compaction_threshold to 0 via schema or nodetool * disabling durable_writes to disable the commit log during the bulk = load.=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/10/2012, at 11:55 PM, Matthias Broecheler = wrote: > Follow up question: Is it safe to abort the compactions happening = after node repair? >=20 > On Mon, Oct 15, 2012 at 6:32 PM, Will Martin = wrote: > +1 It doesn't make sense that the xfr compactions are heavy unless = they are translating the file. This could be a protocol mismatch: = however the requirements for node level compaction and wire compaction I = would expect to be pretty different. > On Oct 15, 2012, at 4:42 PM, Matthias Broecheler wrote: >=20 > > Hey, > > > > we are writing a lot of data into a cassandra cluster for a batch = loading use case. We cannot use the sstable batch loader, so in order to = speed up the loading process we are using RF=3D1 while the data is = loading. After the load is complete, we want to increase the RF. For = that, we are updating the RF in the schema and then run the node repair = tool on each cassandra instance to stream the data over. However, we are = noticing that this process is slowed down by a lot of compactions (the = actually streaming of data only takes a couple of minutes). > > > > Cassandra is already running a major compaction after the data = loading process has completed. But then, there are to be two more = compactions (one on the sender and one on the receiver) happening and = those take a very long time even on the aws high i/o instance with no = compaction throttling. > > > > Question: These additional compactions seem redundant since there = are no reads or writes on the cluster after the first major compaction = (immediately after the data load), is that right? And if so, what can we = do to avoid them? We are currently waiting multiple days. > > > > Thank you very much for your help, > > Matthias > > >=20 >=20 >=20 >=20 > --=20 > Matthias Broecheler, PhD > http://www.matthiasb.com > E-Mail: me@matthiasb.com --Apple-Mail=_308CA26B-7F71-4192-BC3B-12D6941739A7 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Follow up question: Is it safe to abort the = compactions happening after node repair?
It is always = safe to abort a compaction. The purpose of compaction is to replicate = the current truth in a more compact format. It does not modify data, it = just creates new files. The worse case would be killing it between the = time the new files are marked as non temp and the time the old files are = deleted. That would result in wasted disk space, but the truth in the = system would not change. 


> Question: These additional = compactions seem redundant since there are no reads or writes on the = cluster after the first major compaction (immediately after the data = load), is that = right?
Repair = transfers a portion of the  -Data.db component from potentially = multiple SSTables. This may result in multiple new SStables being = created on the receiving node. Once the files are created they are = processed in a similar way to when a memtable is flushed and so = compaction kicks in.

And if so, what can we do to avoid = them? We are currently waiting multiple = days.
That fact = that compaction is taking so long is odd. Have you checked the logs for = GC problems? if you are running an SSD backed instance and have turned = off compaction throttling the high IO throughput can result in mucho = garbage. Faster is not always better. 

To = improve your situation consider:

* disabling = compaction by setting min_compaction_threshold and = max_compaction_threshold to 0 via schema or nodetool
* = disabling durable_writes to disable the commit log during the bulk = load. 

Cheers


<= div apple-content-edited=3D"true">
http://www.thelastpickle.com

On 17/10/2012, at 11:55 PM, Matthias Broecheler <me@matthiasb.com> = wrote:

Follow up question: Is it safe to abort the compactions = happening after node repair?

On Mon, = Oct 15, 2012 at 6:32 PM, Will Martin <will@voodoolunchbox.com> wrote:
+1   It doesn't make sense that the xfr compactions are = heavy unless they are translating the file. This could be a protocol = mismatch: however the requirements for node level compaction and wire = compaction I would expect to be pretty different.
On Oct 15, 2012, at 4:42 PM, = Matthias Broecheler wrote:

> Hey,
>
> we are writing a lot of data into a cassandra cluster for a batch = loading use case. We cannot use the sstable batch loader, so in order to = speed up the loading process we are using RF=3D1 while the data is = loading. After the load is complete, we want to increase the RF. For = that, we are updating the RF in the schema and then run the node repair = tool on each cassandra instance to stream the data over. However, we are = noticing that this process is slowed down by a lot of compactions (the = actually streaming of data only takes a couple of minutes).
>
> Cassandra is already running a major compaction after the data = loading process has completed. But then, there are to be two more = compactions (one on the sender and one on the receiver) happening and = those take a very long time even on the aws high i/o instance with no = compaction throttling.
>
> Question: These additional compactions seem redundant since there = are no reads or writes on the cluster after the first major compaction = (immediately after the data load), is that right? And if so, what can we = do to avoid them? We are currently waiting multiple days.
>
> Thank you very much for your help,
> Matthias
>




-- =
Matthias Broecheler, PhD
http://www.matthiasb.com
E-Mail: me@matthiasb.com

= --Apple-Mail=_308CA26B-7F71-4192-BC3B-12D6941739A7--