Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8F1E3444A for ; Sun, 15 May 2011 22:05:17 +0000 (UTC) Received: (qmail 13178 invoked by uid 500); 15 May 2011 22:05:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13152 invoked by uid 500); 15 May 2011 22:05:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13144 invoked by uid 99); 15 May 2011 22:05:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2011 22:05:15 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a82.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2011 22:05:06 +0000 Received: from homiemail-a82.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a82.g.dreamhost.com (Postfix) with ESMTP id 26591282060 for ; Sun, 15 May 2011 15:04:44 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=kjM4vnfbLx Qgronqn7GpcnguGCzCU21VoNHWZpYJ9+e71go6vbM4ksoyTLB4O0Zpf8Xb29BQO+ g2EZq69/XnwhYxBqAnYRavZ3tAbPEYIq8ndtJ8DoACrcBlPEF1VFaP4bMHZZap+A ErSyhnkTTIsWaC5GmioXgyc3oFyQKS15E= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=KyxO/XYGHsHuKzoN 9oOg6quIzTE=; b=fsjb3ac0NQwXyBAZVgEEu07opO8acj6GW/OVqQiXMsxMMYdH HNhooHYomdtC7w0MXzKbNpgkabnhYOYyKjDKrrRWdQg7NWd0nLISF5h6JzvK7VC6 CVuZJ2tWCPG5JV7C0pzjfak/Y/znEhNIwIErSwNlz/L6GPXHmVQGm7lY/u0= Received: from [10.0.1.151] (121-73-157-230.cable.telstraclear.net [121.73.157.230]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a82.g.dreamhost.com (Postfix) with ESMTPSA id 02B1B28205F for ; Sun, 15 May 2011 15:04:42 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-1-772365639 Subject: Re: Import/Export of Schema Migrations Date: Mon, 16 May 2011 10:04:39 +1200 In-Reply-To: To: user@cassandra.apache.org References: <6633D0DF-DD96-40A4-A1D3-714DF123A362@thelastpickle.com> Message-Id: X-Mailer: Apple Mail (2.1084) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-1-772365639 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Not sure what sort of changes you are making, but this is my approach.=20= I've always managed database (my sql, sql server whatever) schema as = source code (SQL DDL statements, CLI script etc). It makes it a lot = easier to cold start the system, test changes and see who changed what.=20= Once you have your initial schema you can hand roll a CLI script to = update / drop existing CF's. For the update column family statement all = the attributes are delta to the current setting, i.e. you do not need to = say comparator is ascii again. Other than the indexes, you need to = specify all the indexes again those not included will be dropped.=20 If you want to be able to replay multiple schema changes made during dev = against other clusters my personal approach would be: - create a cli script for every change (using update and delete CF), = prefixed with 000X so you can see the order.=20 - manage the scripts in source control - sanity check to see if they can be collapsed=20 - replay the changes in order when applying them to a cluster.=20 (you will still need to manually delete data from dropped cf's) =20 changes to conf/cassandra.yaml can be managed using chef=20 =20 Others will have different ideas.... ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 14 May 2011, at 00:15, David Boxenhorn wrote: > Actually, I want a way to propagate *any* changes from development to = staging to production, but schema changes are the most important. >=20 > Could I use 2221 to propagate schema changes by deleting the schema in = the target cluster, doing "show schema" in the source cluster, = redirecting to a file, and running the file as a script in the target = cluster?=20 >=20 > Of course, I would have to delete the files of dropped CFs by hand = (something I *hate* to do, because I'm afraid of making a mistake), but = it would be a big improvement.=20 >=20 > I am open to any other ideas of how to propagate changes from one = cluster to another in an efficient non-error-prone fashion. Our = development environment (i.e. development, staging, production) is = pretty standard, so I'm sure that I'm not the only one with this = problem!=20 >=20 >=20 > On Fri, May 13, 2011 at 12:51 PM, aaron morton = wrote: > What sort of schema changes are you making? can you manage them as a = CLI script under source control ?=20 >=20 >=20 > You may also be interested in CASSANDRA-2221. >=20 > Cheers > Aaron > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com >=20 > On 12 May 2011, at 20:45, David Boxenhorn wrote: >=20 >> My use case is like this: I have a development cluster, a staging = cluster and a production cluster. When I finish a set of migrations = (i.e. changes) on the development cluster, I want to apply them to the = staging cluster, and eventually the production cluster. I don't want to = do it by hand, because it's a painful and error-prone process. What I = would like to do is export the last N migrations from the development = cluster as a text file, with exactly the same format as the original = text commands, and import them to the staging and production clusters. >>=20 >> I think the best place to do this might be the CLI, since you would = probably want to view your migrations before exporting them. Something = like this: >>=20 >> show migrations N; Shows the last N migrations. >> export migrations N ; Exports the last N migrations = to file fileName. >> import migrations ; Imports migrations from = fileName. >>=20 >> The import process would apply the migrations one at a time giving = you feedback like, "applying migration: update column family...". If a = migration fails, the process should give an appropriate message and = stop. >>=20 >> Is anyone else interested in this? I have created a Jira ticket for = it here: >>=20 >> https://issues.apache.org/jira/browse/CASSANDRA-2636=20 >>=20 >>=20 >=20 >=20 --Apple-Mail-1-772365639 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
http://www.thelastpickle.com

On 14 May 2011, at 00:15, David Boxenhorn wrote:

Actually, I want a way to propagate *any* changes from = development to staging to production, but schema changes are the most = important.

Could I use 2221 to propagate schema changes by = deleting the schema in the target cluster, doing "show schema" in the = source cluster, redirecting to a file, and running the file as a script = in the target cluster?

Of course, I would have to delete the files of dropped CFs by hand = (something I *hate* to do, because I'm afraid of making a mistake), but = it would be a big improvement.

I am open to any other ideas of = how to propagate changes from one cluster to another in an efficient = non-error-prone fashion. Our development environment (i.e. development, = staging, production) is pretty standard, so I'm sure that I'm not the = only one with this problem!


On Fri, May 13, 2011 at 12:51 PM, = aaron morton <aaron@thelastpickle.com> wrote:
What sort of schema changes are you = making?  can you manage them as a CLI script under source control = ? 


You may also be interested in =  CASSANDRA-2221.

Cheers
Aaron
-----------------
Aaron Morton
Freelance = Cassandra Developer
@aaronmorton

On 12 May 2011, at 20:45, David Boxenhorn = wrote:

My use = case is like this: I have a development cluster, a staging cluster and a = production cluster. When I finish a set of migrations (i.e. changes) on = the development cluster, I want to apply them to the staging cluster, = and eventually the production cluster. I don't want to do it by hand, = because it's a painful and error-prone process. What I would like to do = is export the last N migrations from the development cluster as a text = file, with exactly the same format as the original text commands, and = import them to the staging and production clusters.

I think the best place to do this might be the CLI, since you would = probably want to view your migrations before exporting them. Something = like this:

show = migrations = N;            =         Shows the last N = migrations.
export migrations N = <fileName>;       Exports the last N = migrations to file fileName.
import = migrations <fileName>;         = Imports migrations from fileName.

The import process would apply the migrations one at a time giving = you feedback like, "applying migration: update column family...". If a = migration fails, the process should give an appropriate message and = stop.

Is anyone else interested in this? I have created a Jira ticket for = it here:

https://issues.apache.org/jira/browse/CASSANDRA-2636=


=



= --Apple-Mail-1-772365639--