Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B8E0318191 for ; Tue, 2 Feb 2016 19:06:49 +0000 (UTC) Received: (qmail 51697 invoked by uid 500); 2 Feb 2016 19:06:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 51655 invoked by uid 500); 2 Feb 2016 19:06:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51645 invoked by uid 99); 2 Feb 2016 19:06:44 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2016 19:06:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A117618027C for ; Tue, 2 Feb 2016 19:06:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.381 X-Spam-Level: ** X-Spam-Status: No, score=2.381 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2.5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id XLvMQtFSRE1O for ; Tue, 2 Feb 2016 19:06:32 +0000 (UTC) Received: from mail-yw0-f177.google.com (mail-yw0-f177.google.com [209.85.161.177]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 97D0120271 for ; Tue, 2 Feb 2016 19:06:31 +0000 (UTC) Received: by mail-yw0-f177.google.com with SMTP id z185so1400416ywf.0 for ; Tue, 02 Feb 2016 11:06:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=aCCwJq9wMMFha6OVPeznWP1PjpeIzjuZ+pct95QqxdA=; b=Kt+bRjyZcsXZVg/4QrE/Ms7Ec+0j66/u68GPwG6KYHoa13+pIA/qEmdWQiwqSydqX+ nEV8AQZAPbyxf9tgmc+oROSlJRHbtBzwe3V7mocusmQLrconjDEAiH6sU2GZiGLI1okm Ewzk6Gaj3nHiA23wR+Pgd8KtD2qQFwnastSo2GYXwFvd0eG3KyVDMjCB0h5tPRfJwOwQ GhUiZjgcIqN1OuABQCE4vVhFttCRedlW/0IBDDGR36ClFxXuW7arjjHMuhSzL9LJTSdr SvyVL852hcH8S5kC6mrzM+5QW9PcYjxaAUKNBoAJoPXorYWK/2svFK+PEcTRsSb4tJ0/ Rraw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=aCCwJq9wMMFha6OVPeznWP1PjpeIzjuZ+pct95QqxdA=; b=YKAKschZIkn0ZYeZjAvIoY6P0SEtlfYkeLYiF/ax17qe3ba/8eL/eEsOvJQhfKh5TC EO8W2Y+dn30PyVZQpSpfPoOaYudaGAjkeL1heSzOGpghFy42Xi+//KYvOuquBtPPMk90 fCxP4rBjvA74TuILR/XR/qDXSB7yqEylZv1sB8lHw6ZFBGR6U7pVfjHzSZ3Pp0DL3bQ5 Ks+yufgfGO+lAuBfc6ZiLYpM+gC86uQoJJXheAld1CKfE8eGgkzVQPM4m5JCdIXZyIHe w7rt2FTOjjJHvD+a+dlg+KVZ0Sw6hnsKVjoY1z4eUDk2mEgkdXBI5VoekHIC5XLpeGqF Sg/w== X-Gm-Message-State: AG10YOQXuC4jmas0Mw36r6Mxvi7YaX5+ZNgOAXws45d4m4rvxUsu0YpL8HMAUmFc/+0uhlVhpkTlJ/p+DHF26A== X-Received: by 10.13.203.216 with SMTP id n207mr12273460ywd.204.1454439984410; Tue, 02 Feb 2016 11:06:24 -0800 (PST) MIME-Version: 1.0 Received: by 10.129.83.214 with HTTP; Tue, 2 Feb 2016 11:06:05 -0800 (PST) In-Reply-To: <1305982805.559485.1454439718249.JavaMail.yahoo@mail.yahoo.com> References: <1305982805.559485.1454439718249.JavaMail.yahoo@mail.yahoo.com> From: DuyHai Doan Date: Tue, 2 Feb 2016 20:06:05 +0100 Message-ID: Subject: Re: Moving Away from Compact Storage To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a114e68f09e3efe052ace325c --001a114e68f09e3efe052ace325c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You 'll need to do the transformation in Spark, although I don't understand what you mean by "dynamic columns". Given the CREATE TABLE script you gave earlier, there is nothing such as dynamic columns On Tue, Feb 2, 2016 at 8:01 PM, Anuj Wadehra wrote= : > Will it be possible to read dynamic columns data from compact storage and > trasform them as collection e.g. map in new table? > > > Thanks > Anuj > > Sent from Yahoo Mail on Android > > > On Wed, 3 Feb, 2016 at 12:28 am, DuyHai Doan > wrote: > So there is no "static" (in the sense of CQL static) column in your legac= y > table. > > Just define a Scala case class to match this table and use Spark to dump > the content to a new non compact CQL table > > On Tue, Feb 2, 2016 at 7:55 PM, Anuj Wadehra > wrote: > >> Our old table looks like this from cqlsh: >> >> CREATE TABLE table table1 ( >> key text, >> "Col1" blob, >> "Col2" text, >> "Col3" text, >> "Col4" text, >> PRIMARY KEY (key) >> ) WITH COMPACT STORAGE AND =E2=80=A6 >> >> And it will have some dynamic text data which we are planning to add in >> collections.. >> >> Please let me know if you need more details.. >> >> >> Thanks >> Anuj >> Sent from Yahoo Mail on Android >> >> >> On Wed, 3 Feb, 2016 at 12:14 am, DuyHai Doan >> wrote: >> Can you give the CREATE TABLE script for you old compact storage table ? >> Or at least the cassandra-client creation script >> >> On Tue, Feb 2, 2016 at 3:48 PM, Anuj Wadehra >> wrote: >> >>> Thanks DuyHai !! We were also thinking to do it the "Spark" way but I >>> was not sure that its would be so simple :) >>> >>> We have a compact storage cf with each row having some data in staticly >>> defined columns while other data in dynamic columns. Is the approach >>> mentioned in link adaptable to the scenario where we want to migrate th= e >>> existing data to a Non-Compact CF with static columns and collections ? >>> >>> Thanks >>> Anuj >>> >>> -------------------------------------------- >>> On Tue, 2/2/16, DuyHai Doan wrote: >>> >>> Subject: Re: Moving Away from Compact Storage >>> To: user@cassandra.apache.org >>> Date: Tuesday, 2 February, 2016, 12:57 AM >>> >>> Use Apache >>> Spark to parallelize the data migration. Look at this piece >>> of code >>> https://github.com/doanduyhai/Cassandra-Spark-Demo/blob/master/src/main= /scala/usecases/MigrateAlbumsData.scala#L58-L60 >>> If your source and target tables >>> have the SAME structure (except for the COMPACT STORAGE >>> clause), migration with Spark is a 2 lines of >>> code >>> On Mon, Feb 1, 2016 at 8:14 >>> PM, Anuj Wadehra >>> wrote: >>> Hi >>> Whats the fastest and reliable way >>> to migrate data from a Compact Storage table to Non-Compact >>> storage table? >>> I was not >>> able to find any command for dropping the compact storage >>> directive..so I think migrating data is the only way...any >>> suggestions? >>> ThanksAnuj >>> >>> >>> >> > --001a114e68f09e3efe052ace325c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You 'll need to do the transformation in Spark, althou= gh I don't understand what you mean by "dynamic columns". Giv= en the CREATE TABLE script you gave earlier, there is nothing such as dynam= ic columns

O= n Tue, Feb 2, 2016 at 8:01 PM, Anuj Wadehra <anujw_2003@yahoo.co.in= > wrote:
Will it be possibl= e to read dynamic columns data from compact storage and trasform them as co= llection e.g. map in new table?


Thanks
Anuj


On Wed, 3 Feb, 2016 at 12:28 am, DuyHai Doan
So there is= no "static" (in the sense of CQL static) column in your legacy t= able.=C2=A0

Just define a Scala case clas= s to match this table and use Spark to dump the content to a new non compac= t CQL table

<= div class=3D"gmail_quote">On Tue, Feb 2, 2016 at 7:55 PM, Anuj Wadehra <anujw_2003@yahoo.co.in= > wrote:
Our= old table looks like this from cqlsh:

CREATE TABLE table table1 (
=C2=A0 key text,
=C2=A0 = "Col1" blob,
=C2=A0 "Col2" text,
= =C2=A0 "Col3" text,
=C2=A0 "Col4" text,
=
=C2=A0 PRIMARY KEY (key)
) WITH COMPACT STORAGE AND =E2=80= =A6

And it will have some dynamic t= ext data which we are planning to add in collections..

Please let me know if you need more details..


Thanks
<= div>Anuj

On Wed, 3 Feb, 2016 at 12:14 am, DuyHai Doan
= <doanduyhai@gmail.com> wrote:<= /div>
Can you give the CREATE TABLE = script for you old compact storage table ? Or at least the cassandra-client= creation script

On Tue, Feb 2, 2016 at 3:48 PM, Anuj Wadehra <anujw_2003@yahoo.co.in<= /a>> wrote:
Than= ks DuyHai !! We were also thinking to do it the "Spark" way but I= was not sure that its would be so simple :)

We have a compact storage cf with each row having some data in staticly def= ined columns while other data in dynamic columns. Is the approach mentioned= in link adaptable to the scenario where we want to migrate the existing da= ta to a Non-Compact CF with static columns and collections ?

Thanks
Anuj

--------------------------------------------
On Tue, 2/2/16, DuyHai Doan <doanduyh= ai@gmail.com> wrote:

=C2=A0Subject: Re: Moving Away from Compact Storage
=C2=A0To: user@cassandra.apache.org<= br clear=3D"none"> =C2=A0Date: Tuesday, 2 February, 2016, 12:57 AM

=C2=A0Use Apache
=C2=A0Spark to parallelize the data migration. Look at this piece
=C2=A0of code=C2=A0https://github.com/doandu= yhai/Cassandra-Spark-Demo/blob/master/src/main/scala/usecases/MigrateAlbums= Data.scala#L58-L60
=C2=A0If your source and target tables
=C2=A0have the SAME structure (except for the COMPACT STORAGE
=C2=A0clause), migration with Spark is a 2 lines of
=C2=A0code
=C2=A0On Mon, Feb 1, 2016 at 8:14
=C2=A0PM, Anuj Wadehra <anujw_2003@ya= hoo.co.in>
=C2=A0wrote:
=C2=A0Hi
=C2=A0Whats the fastest and reliable way
=C2=A0to migrate data from a Compact Storage table to Non-Compact
=C2=A0storage table?
=C2=A0I was not
=C2=A0able to find any command for dropping the compact storage
=C2=A0directive..so I think migrating data is the only way...any
=C2=A0suggestions?
=C2=A0ThanksAnuj





--001a114e68f09e3efe052ace325c--