Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 19FB78D1A for ; Thu, 11 Aug 2011 02:00:16 +0000 (UTC) Received: (qmail 80170 invoked by uid 500); 11 Aug 2011 02:00:13 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 80148 invoked by uid 500); 11 Aug 2011 02:00:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 80139 invoked by uid 99); 11 Aug 2011 02:00:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Aug 2011 02:00:12 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a53.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Aug 2011 02:00:05 +0000 Received: from homiemail-a53.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a53.g.dreamhost.com (Postfix) with ESMTP id 9F611138059 for ; Wed, 10 Aug 2011 18:59:42 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=Ss+JBXpTFZ U/4PIOsyXlckc73veuHVw8J3Cu38FBrXf/HyK/HNBKuUgE7MmtFhDjQodl9A4e6O E9ZNtrB3RyC1fV0OFrB6i1CKmyPdSb+VpQkh7ryOvk5RwqR569QUQ86+0IdYM5rz SeiqGjjMH6VL5ChdtAOzlr5VjFLc1BSDk= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=DoiJEr437BPJC3Df tMyaTPFG1Hw=; b=3tvXolOeUVs/XI2aAnAet+7HS1VYF+cv30sg4UKqWY3rNB/r pPnx6epRR3gjKMjvC8AOa1S+0rj3Q/kfTXnRb7xDwzeuHnUOZY2ajNEK38olCxV0 DroV0EAV9HIYeHWbFX11QcW3DkiANMkcNKlpCSSfktduOnqtJxfRT93kPxI= Received: from 202-126-206-202.vectorcommunications.net.nz (unknown [202.126.206.202]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a53.g.dreamhost.com (Postfix) with ESMTPSA id 30433138058 for ; Wed, 10 Aug 2011 18:59:42 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_1E069596-30B8-47C1-B126-E0B96B1F30D9" Subject: Re: Bootstrapping Date: Thu, 11 Aug 2011 13:59:38 +1200 In-Reply-To: <8E8C8608-59F7-43F7-BEF1-94751F99DFAE@gmail.com> To: user@cassandra.apache.org References: <8E8C8608-59F7-43F7-BEF1-94751F99DFAE@gmail.com> Message-Id: X-Mailer: Apple Mail (2.1244.3) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_1E069596-30B8-47C1-B126-E0B96B1F30D9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii First, upgrade from 0.7.5 if possible. This is as good a reason as any = https://github.com/apache/cassandra/blob/cassandra-0.7.8/CHANGES.txt#L58 Can you copy the SSTables off node and then just bring it back ? It will = be *a lot* faster than use nodetool repair. (drain the node first to = clear the commit log). Or if you have a spare machine perform a rolling = migration. If at all possible I would try to do it as an upgrade described above. = It will be much much easier.=20 If you plan to turn a node off and clear it's data you should remove the = nodes token from the ring. You can either use nodetool decommission = which will distribute the data around the ring, or turn it off and then = use nodetool remove token which will not.=20 > 1. I realize this will put a heavier I/O load on the replication nodes = to AntiCompact the CF's, but what kind of load does this put on the JVM. = Are there any gotchas I should be aware of to prevent long gc times or = OOM exceptions on the replication nodes. We don't have the AnitCompaction step any more. If your app is stable I = would assume the repair process would be to. Do your normal repair = processed complete ok ? > 3. Documentation at http://wiki.apache.org/cassandra/Operations says = that the thrift port is not active on the bootstrapping node during the = streaming process. What is the process that brings the node up-to-date = with mutations that occurred during the time of the bootstrap? Maybe = it's only reads that are disabled and writes are allowed? Thrift is the connection the client uses, disabling it means clients = cannot write to it. The node will announce it's intention to take = ownership of a token range in the ring when the bootstrap starts. =46rom = that point on other nodes will include it in write requests but not read = requests. During that time your data is replicated to RF+1 nodes.=20 =20 > 4. What happens if schema changes (add/drop column families) occur in = the cluster while the bootstrap is in progress? They will be distributed to the node when it comes back. Until it gets = the new updates it will log ERRORs for mutations to non existent CF's. = Best advice is do not make those changes.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 11 Aug 2011, at 09:54, Chad Johnson wrote: > Hi, >=20 > I have a 15 node cluster with a RF=3D3 running version 0.7.5. I am = planning to perform some filesystem maintenance on each of the nodes. = The filesystem happens to be on the partition holding the keyspace data. = The maintenance means that all the SSTables for our keyspace will be = destroyed. Rather than backup all the data to a backup disk and restore, = my plan was to bring the node down, perform the maintenance, keep the = original initial_token, set auto_bootstrap to true and let Cassandra = repopulate the data through the streaming process. Nodes in the cluster = will have a load of about 250 to 300GB >=20 > I have a couple questions regarding bootstrapping and the streaming = process. >=20 > 1. I realize this will put a heavier I/O load on the replication nodes = to AntiCompact the CF's, but what kind of load does this put on the JVM. = Are there any gotchas I should be aware of to prevent long gc times or = OOM exceptions on the replication nodes. > 2. If the initial_token is not changed, is it correct to assume that = anticompaction will occur only on the replication nodes and not = throughout the cluster as the key space has not been modified. > 3. Documentation at http://wiki.apache.org/cassandra/Operations says = that the thrift port is not active on the bootstrapping node during the = streaming process. What is the process that brings the node up-to-date = with mutations that occurred during the time of the bootstrap? Maybe = it's only reads that are disabled and writes are allowed? > 4. What happens if schema changes (add/drop column families) occur in = the cluster while the bootstrap is in progress? >=20 > Thanks for your help >=20 > Chad --Apple-Mail=_1E069596-30B8-47C1-B126-E0B96B1F30D9 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii https://github.com/apache/cassandra/blob/cassandra-0.7.8/CHANGES.t= xt#L58

Can you copy the SSTables off node and = then just bring it back ? It will be *a lot* faster than use nodetool = repair. (drain the node first to clear the commit log). Or if you have a = spare machine perform a rolling migration.

If = at all possible I would try to do it as an upgrade described above. It = will be much much easier. 

If you plan to = turn a node off and clear it's data you should remove the nodes token = from the ring. You can either use nodetool decommission which will = distribute the data around the ring, or turn it off and then use = nodetool remove token which will = not. 

1. I realize this will put a heavier I/O load on the = replication nodes to AntiCompact the CF's, but what kind of load does = this put on the JVM. Are there any gotchas I should be aware of to = prevent long gc times or OOM exceptions on the replication = nodes.
We don't have the AnitCompaction step any = more. If your app is stable I would assume the repair process would be = to. Do your normal repair processed complete ok = ?

3. = Documentation at http://wiki.apache.or= g/cassandra/Operations says that the thrift port is not active on = the bootstrapping node during the streaming process. What is the process = that brings the node up-to-date with mutations that occurred during the = time of the bootstrap? Maybe it's only reads that are disabled and = writes are allowed?
Thrift is the = connection the client uses, disabling it means clients cannot write to = it. The node will announce it's intention to take ownership of a token = range in the ring when the bootstrap starts. =46rom that point on other = nodes will include it in write requests but not read requests. During = that time your data is replicated to RF+1 = nodes. 
 
4. = What happens if schema changes (add/drop column families) occur in the = cluster while the bootstrap is in progress?
They = will be distributed to the node when it comes back. Until it gets the = new updates it will log ERRORs for mutations to non existent CF's. Best = advice is do not make those = changes. 

Cheers

http://www.thelastpickle.com

On 11 Aug 2011, at 09:54, Chad Johnson wrote:

Hi,

I have a 15 node cluster with a RF=3D3 = running version 0.7.5. I am planning to perform some filesystem = maintenance on each of the nodes. The filesystem happens to be on the = partition holding the keyspace data. The maintenance means that all the = SSTables for our keyspace will be destroyed. Rather than backup all the = data to a backup disk and restore, my plan was to bring the node down, = perform the maintenance, keep the original initial_token, set = auto_bootstrap to true and let Cassandra repopulate the data through the = streaming process. Nodes in the cluster will have a load of about 250 to = 300GB

I have a couple questions regarding bootstrapping and the = streaming process.

1. I realize this will put a heavier I/O load = on the replication nodes to AntiCompact the CF's, but what kind of load = does this put on the JVM. Are there any gotchas I should be aware of to = prevent long gc times or OOM exceptions on the replication nodes.
2. = If the initial_token is not changed, is it correct to assume that = anticompaction will occur only on the replication nodes and not = throughout the cluster as the key space has not been modified.
3. = Documentation at http://wiki.apache.or= g/cassandra/Operations says that the thrift port is not active on = the bootstrapping node during the streaming process. What is the process = that brings the node up-to-date with mutations that occurred during the = time of the bootstrap? Maybe it's only reads that are disabled and = writes are allowed?
4. What happens if schema changes (add/drop = column families) occur in the cluster while the bootstrap is in = progress?

Thanks for your = help

Chad

= --Apple-Mail=_1E069596-30B8-47C1-B126-E0B96B1F30D9--