Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B497164A0 for ; Thu, 4 Aug 2011 03:07:41 +0000 (UTC) Received: (qmail 28571 invoked by uid 500); 4 Aug 2011 03:07:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 28260 invoked by uid 500); 4 Aug 2011 03:07:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28247 invoked by uid 99); 4 Aug 2011 03:07:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Aug 2011 03:07:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-ew0-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Aug 2011 03:07:21 +0000 Received: by ewy19 with SMTP id 19so768570ewy.31 for ; Wed, 03 Aug 2011 20:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=iJvXYpYLsxI0SnPrS4hfMtcbkg5geDnwrcN+1aQcaGY=; b=Sh8YtziJSvL5ylHBYgihaSXRRQqx4FJgWa7pg4+KT5TMolFY5BQnAfQdBtD6R+QO/j cc4Ay+chShbukoV1nOPyP9RZRwab6+4H+2FNwenY9i0PNSBFhqi7LdRq9mPYGCBPivSX T7TSfKUlNrRdTDozhvS+4vVXBq3JB/N6U+XbI= Received: by 10.213.110.1 with SMTP id l1mr108616ebp.109.1312427220181; Wed, 03 Aug 2011 20:07:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.108.68 with HTTP; Wed, 3 Aug 2011 20:06:40 -0700 (PDT) In-Reply-To: <1312424740.29615.41.camel@greenlantern.local> References: <1312333592.29615.11.camel@greenlantern.local> <1312340402.29615.31.camel@greenlantern.local> <1312424740.29615.41.camel@greenlantern.local> From: Jonathan Ellis Date: Wed, 3 Aug 2011 22:06:40 -0500 Message-ID: Subject: Re: Input on difficult migration issue from 0.7.4 to 0.8.2 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0ce0ed5440a44b04a9a54615 X-Virus-Checked: Checked by ClamAV on apache.org --000e0ce0ed5440a44b04a9a54615 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You'd have to check ColumnFamilyStore.open to see for sure. It might work that way, but if it does it's an accident -- seems totally fair for it to assume that there are no SSTables for a newly created CF. On Wed, Aug 3, 2011 at 9:25 PM, Todd Nine wrote: > ** > Hi Johnathan, > One more question. I'm experimenting with the chef recipes for brisk. > Upon the initial installation, the brisk recipe removes the existing syst= em > Keyspace if it's present. The new cluster will not have the" Spidertrac= ks" > keyspace in it's meta data, and will be a completely vanilla installation= . > > Can I leave the SSTable files in place, and create the keyspace and colum= n > families via the CLI without needing to restart the whole cluster? I.E, > when a CF is added to the schema, if an SSTable for that CF exists, is it > merged with the commitlog in the next compaction, and used for reads or i= s > it ignored? > > Thanks, > Todd > > > > On Tue, 2011-08-02 at 22:17 -0500, Jonathan Ellis wrote: > > Yes, that should work. For better/worse, CF metadata is stored in the > system schema, not individual sstables. > > On Tue, Aug 2, 2011 at 10:00 PM, Todd Nine wrote: > > Hi Johnathan. Are these the correct steps for the upgrade? > > > > 1. Copy SSTable from 0.7.4 snapshot > > 2. Drop CFS in all nodes in 0.7.4 cluster > > 3. Upgrade to 0.8.2 and start cluster (with no clients connected) > > 4. Create CFs using cli in cluster > > 5. Shutdown all nodes > > 6. Move the SSTables into place on all the nodes > > 7. Restart the nodes > > > > This would definitely be my preferred approach, however I wasn't sure i= f > > SSTables stored the CF meta data. > > > > Thanks, > > Todd > > > > > > On Tue, 2011-08-02 at 21:34 -0500, Jonathan Ellis wrote: > > > > If the bytes are compatible why not just use the existing sstable > > files after redefining the schema? > > > > On Tue, Aug 2, 2011 at 8:06 PM, Todd Nine wrote= : > >> Hi all, > >> I seem to have backed myself into a corner and I cannot easily upgra= de > >> from a custom 0.7.4 installation to the default 0.8.2. Any help I cou= ld > >> get > >> would be greatly appreciated. Below is an outline of the problem. > >> > >> Current installation: > >> 0.7.4 with Ed Anuff's custom composite comparators. > >> > >> https://github.com/riptano/hector-composite/commit/74407df3ef97c3fa90e= ae9f68d84b50be3d9938e > >> CF Comprator Type : "DynamicComposite" > >> > >> > >> New installation: > >> 0.8.2 with Casasndra's native DynamicComposite column time. > >> CF Comparator Type: > >> > >> "DynamicComposite(a=3D>AsciiType,b=3D>BytesType,i=3D>IntegerType,x=3D>= LexicalUUIDType,l=3D>LongType,t=3D>TimeUUIDType,s=3D>UTF8Type,u=3D>UUIDType= ,A=3D>AsciiType(reversed=3Dtrue),B=3D>BytesType(reversed=3Dtrue),I=3D>Integ= erType(reversed=3Dtrue),X=3D>LexicalUUIDType(reversed=3Dtrue),L=3D>LongType= (reversed=3Dtrue),T=3D>TimeUUIDType(reversed=3Dtrue),S=3D>UTF8Type(reversed= =3Dtrue),U=3D>UUIDType(reversed=3Dtrue))" > >> > >> > >> I cannot simply upgrade Cassandra, this fails because the comparator i= s > >> incorrectly defined for version 0.8.1 and on. My issue is that the co= lumn > >> family definition has changed, the bytes that are stored are compatibl= e, I > >> simply need to re-define the CF and migrate the column data over. > >> > >> > >> Initially I was thinking I could perform a read from the rows and colu= mns > >> from a single 0.7 node, then insert them into the 0.8 cluster, however= I > >> cannot have 2 different versions of the thrift API running in the same > >> java > >> JVM due to namespace conflicts. > >> > >> > >> Idea is to perform the following steps, if anyone has any better > >> suggestions, they would be greatly appreciated. > >> > >> 1. Back up all my CFs that use the dynamic composite, and copy the SST= able > >> out of my Keyspace's data directory to tmp > >> 2. Drop all CFs that use dynamic composite > >> 3. Re-create the CFs with the new comparator definition > >> 4. Using an external program directly read the 0.7 SSTables (without > >> starting the cassandra daemon) and insert the rows and columns into th= e > >> 0.8 > >> cluster via thrift. > >> > >> > >> Can anyone point me at a good example for reading rows and columns > >> directly > >> from the SSTables without using the thrift api? > >> > >> Thanks, > >> Todd > > > > > > > > > > > > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com --000e0ce0ed5440a44b04a9a54615 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You'd have to check ColumnFamilyStore.open to see for sure.=A0 It might= work that way, but if it does it's an accident -- seems totally fair f= or it to assume that there are no SSTables for a newly created CF.

On Wed, Aug 3, 2011 at 9:25 PM, Todd Nine <todd@spidertracks.= com> wrote:
=20 =20
Hi Johnathan,
=A0 One more question.=A0 I'm experimenting with the chef recipes for b= risk.=A0 Upon the initial installation, the brisk recipe removes the existi= ng system Keyspace if it's present.=A0=A0 The new cluster will not have= the" Spidertracks" keyspace in it's meta data, and will be a= completely vanilla installation.=A0

Can I leave the SSTable files in place, and create the keyspace and column = families via the CLI without needing to restart the whole cluster?=A0 I.E, = when a CF is added to the schema, if an SSTable for that CF exists, is it m= erged with the commitlog in the next compaction, and used for reads or is i= t ignored?

Thanks,
Todd
On Tue, 2011-08-02 at 22:17 -0500, Jonathan Ellis wrote: 
Yes, that should work.  For better/worse, CF metadata is stored in the
system schema, not individual sstables.

On Tue, Aug 2, 2011 at 10:00 PM, Todd Nine <todd@spidertracks.com> wrote:
> Hi Johnathan.=A0 Are these the correct steps for the upgrade?
>
> 1. Copy SSTable from 0.7.4 snapshot
> 2. Drop CFS in all nodes in 0.7.4 cluster
> 3. Upgrade to 0.8.2 and start cluster (with no clients connected)
> 4. Create CFs using cli in cluster
> 5. Shutdown all nodes
> 6. Move the SSTables into place on all the nodes
> 7. Restart the nodes
>
> This would definitely be my preferred approach, however I wasn't s=
ure if
> SSTables stored the CF meta data.
>
> Thanks,
> Todd
>
>
> On Tue, 2011-08-02 at 21:34 -0500, Jonathan Ellis wrote:
>
> If the bytes are compatible why not just use the existing sstable
> files after redefining the schema?
>
> On Tue, Aug 2, 2011 at 8:06 PM, Todd Nine <todd@spidertracks.com> wrote:
>> Hi all,
>> =A0 I seem to have backed myself into a corner and I cannot easily=
 upgrade
>> from a custom 0.7.4 installation to the default 0.8.2.=A0 Any help=
 I could
>> get
>> would be greatly appreciated.=A0 Below is an outline of the proble=
m.
>>
>> Current installation:
>> 0.7.4 with Ed Anuff's custom composite comparators.
>>
>> https://github.com/=
riptano/hector-composite/commit/74407df3ef97c3fa90eae9f68d84b50be3d9938e
>> CF Comprator Type : "DynamicComposite"
>>
>>
>> New installation:
>> 0.8.2 with Casasndra's native DynamicComposite column time.
>> CF Comparator Type:
>>
>> "DynamicComposite(a=3D>AsciiType,b=3D>BytesType,i=3D>=
;IntegerType,x=3D>LexicalUUIDType,l=3D>LongType,t=3D>TimeUUIDType,=
s=3D>UTF8Type,u=3D>UUIDType,A=3D>AsciiType(reversed=3Dtrue),B=3D&g=
t;BytesType(reversed=3Dtrue),I=3D>IntegerType(reversed=3Dtrue),X=3D>L=
exicalUUIDType(reversed=3Dtrue),L=3D>LongType(reversed=3Dtrue),T=3D>T=
imeUUIDType(reversed=3Dtrue),S=3D>UTF8Type(reversed=3Dtrue),U=3D>UUID=
Type(reversed=3Dtrue))"
>>
>>
>> I cannot simply upgrade Cassandra, this fails because the comparat=
or is
>> incorrectly defined for version 0.8.1 and on.=A0 My issue is that =
the column
>> family definition has changed, the bytes that are stored are compa=
tible, I
>> simply need to re-define the CF and migrate the column data over.
>>
>>
>> Initially I was thinking I could perform a read from the rows and =
columns
>> from a single 0.7 node, then insert them into the 0.8 cluster, how=
ever I
>> cannot have 2 different versions of the thrift API running in the =
same
>> java
>> JVM due to namespace conflicts.
>>
>>
>> Idea is to perform the following steps, if anyone has any better
>> suggestions, they would be greatly appreciated.
>>
>> 1. Back up all my CFs that use the dynamic composite, and copy the=
 SSTable
>> out of my Keyspace's data directory to tmp
>> 2. Drop all CFs that use dynamic composite
>> 3. Re-create the CFs with the new comparator definition
>> 4. Using an external program directly read the 0.7 SSTables (witho=
ut
>> starting the cassandra daemon) and insert the rows and columns int=
o the
>> 0.8
>> cluster via thrift.
>>
>>
>> Can anyone point me at a good example for reading rows and columns
>> directly
>> from the SSTables without using the thrift api?
>>
>> Thanks,
>> Todd
>
>
>
>



--
Jonathan Ellis
Proje= ct Chair, Apache Cassandra
co-founder of DataStax, the source for profes= sional Cassandra support
http://www.datastax.com
--000e0ce0ed5440a44b04a9a54615--