Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E1DB010597 for ; Thu, 20 Feb 2014 19:32:15 +0000 (UTC) Received: (qmail 8571 invoked by uid 500); 20 Feb 2014 19:32:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 8532 invoked by uid 500); 20 Feb 2014 19:32:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 8522 invoked by uid 99); 20 Feb 2014 19:32:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Feb 2014 19:32:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=AC_DIV_BONANZA,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com designates 74.125.82.43 as permitted sender) Received: from [74.125.82.43] (HELO mail-wg0-f43.google.com) (74.125.82.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Feb 2014 19:32:07 +0000 Received: by mail-wg0-f43.google.com with SMTP id a1so1822705wgh.22 for ; Thu, 20 Feb 2014 11:31:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hw/PxNPyw8Bu90F34A3UhT0++aX2pjQ9dro+WXgopds=; b=NPCO1rbtGCJG6Cut0/vHrBJRhEOImVrGBuZrN8dHuOI0smnFdVO3JpD03tBXM/C2NX 7F7Ljgoa8SYiahauyHex8xrrPURNGyrOXZTh2QNrPA9+GQIh5Zab7G5S8spBIKI89WAR mNhVMsVcNgZ53HGORer0aIB+UnuIz1Sge/iIAD/4A7vvdoww/GpQU+KPUIyZlMMhramt juCf4rErRHlezmLRxpHMviDBcmvc0By43e93Sl3Z1WVZoibItzGOhr3xT/j3W18bOGQf 9RAvuW5qg4p3S0Zr7Zsn22dj0blS71d3FCr4LGVK2Q4n4/zA6VYj2SQAUKTHqX5BpiIt PszQ== MIME-Version: 1.0 X-Received: by 10.194.119.168 with SMTP id kv8mr3660040wjb.41.1392924706160; Thu, 20 Feb 2014 11:31:46 -0800 (PST) Received: by 10.194.220.105 with HTTP; Thu, 20 Feb 2014 11:31:46 -0800 (PST) In-Reply-To: References: Date: Thu, 20 Feb 2014 14:31:46 -0500 Message-ID: Subject: Re: Performance problem with large wide row inserts using CQL From: Edward Capriolo To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=089e01176b1d4f480104f2db8eb3 X-Virus-Checked: Checked by ClamAV on apache.org --089e01176b1d4f480104f2db8eb3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The only thing you really can not do CQL3 loses some of the concept of CQL2 metadata, namedly the default validation and then column specific validation. In cassandra-cql we can say (butchering the syntax) create column family x DEFAULT_VALIDATOR =3D UTF8Type columns named y are int columns named z are string You can do this in CQL.: create table x ( rowkey blob, column blob, value blob, primary key(rowkey,column) using compact storage ; But you lost the concept of columns named y validate as int. Everything is just blob as far as CQL understands it. That being said in the schema presented nothing stops the user from implementing their design in either "system" On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne wr= ote: > On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin wrote: > >> >> I disagree with the sentiment that "thrift is not worth the trouble". >> > > Way to quote only part of my sentence and get mental on it. My full > sentence was "it's probably not worth the trouble to start with thrift if > you're gonna use CQL later". > > >> >> CQL and all SQL inspired dialects limit one's ability to use arbitrary >> typed data in dynamic columns. With thrift it's easy and straight forwar= d. >> With CQL there is no way to tell Cassandra the type of the name and valu= e >> for a dynamic column. You can only set the default type. That means usin= g a >> "pure cql" approach you can deviate from the default type. Cassandra wil= l >> throw an exception indicating the type is different than the default typ= e. >> > >> Until such time that CQL abandons the shackles of SQL and adds the >> ability to indicate the column and value type. Something like this >> > >> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int, >> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as >> long) >> >> This is one area where Thrift is superior to CQL. Having said that, it's >> valid to use Cassandra "as if" it was a relational database, but then yo= u'd >> miss out on some of the unique features. >> > > Man, if I had a nickel every time someone came on that mailing list > pretending that something was possible with thrift and not CQL ... I will > claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, ther= e > is *nothing* that thrift can do that CQL cannot. But well, what do I know > about Cassandra. > > -- > Sylvain > > > >> >> >> >> >> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne wrote: >> >>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo wrote: >>> >>>> For what it is worth you schema is simple and uses compact storage. >>>> Thus you really dont need anything in cassandra 2.0 as far as i can te= ll. >>>> You might be happier with a stable release like 1.2.something and just >>>> hector or astyanax. You are really dealing with many issues you should= not >>>> have to just to protoype a simple cassandra app. >>> >>> >>> >>> Of course, if everyone was using that reasoning, no-one would ever test >>> new features and report problems/suggest improvement. So thanks to anyo= ne >>> like R=FCdiger that actually tries stuff and take the time to report pr= oblems >>> when they think they encounter one. Keep at it, *you* are the one helpi= ng >>> Cassandra to get better everyday. >>> >>> And you are also right R=FCdiger that it's probably not worth the troub= le >>> to start with thrift if you're gonna use CQL later. And you definitivel= y >>> should use CQL, it is Cassandra's future. >>> >>> -- >>> Sylvain >>> >>> >>> >>>> >>>> On Thursday, February 20, 2014, Sylvain Lebresne >>>> wrote: >>>> > >>>> > >>>> > >>>> > On Wed, Feb 19, 2014 at 9:38 PM, R=FCdiger Klaehn >>>> wrote: >>>> >> >>>> >> I have cloned the cassandra repo, applied the patch, and built it. >>>> But when I want to run the bechmark I get an exception. See below. I t= ried >>>> with a non-managed dependency to >>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, wh= ich I >>>> compiled from source because I read that that might help. But that did= not >>>> make a difference. >>>> >> >>>> >> So currently I don't know how to give the patch a try. Any ideas? >>>> >> >>>> >> cheers, >>>> >> >>>> >> R=FCdiger >>>> >> >>>> >> Exception in thread "main" java.lang.IllegalArgumentException: >>>> replicate_on_write is not a column defined in this metadata >>>> >> at >>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions= .java:273) >>>> >> at >>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitio= ns.java:279) >>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117) >>>> >> at >>>> com.datastax.driver.core.TableMetadata$Options.(TableMetadata.ja= va:474) >>>> >> at >>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107) >>>> >> at >>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128= ) >>>> >> at >>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89) >>>> >> at >>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnec= tion.java:259) >>>> >> at >>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnectio= n.java:214) >>>> >> at >>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlCo= nnection.java:161) >>>> >> at >>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.j= ava:77) >>>> >> at >>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890) >>>> >> at >>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910) >>>> >> at >>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806) >>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158) >>>> >> at >>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestM= inimized.scala:31) >>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40) >>>> >> at >>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:1= 2) >>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71) >>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71) >>>> >> at scala.collection.immutable.List.foreach(List.scala:318) >>>> >> at >>>> scala.collection.generic.TraversableForwarder$class.foreach(Traversabl= eForwarder.scala:32) >>>> >> at scala.App$class.main(App.scala:71) >>>> >> at >>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5) >>>> >> at >>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala) >>>> > >>>> > I believe you've tried the cassandra trunk branch? trunk is basicall= y >>>> the future Cassandra 2.1 and the driver is currently unhappy because t= he >>>> replicate_on_write option has been removed in that version. I'm suppos= ed to >>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you'= re >>>> also using a slightly old version of the driver sources in there? Or m= aybe >>>> I've screwed up my fix, I'll double check. But anyway, it would be ove= rall >>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which= you >>>> shouldn't run into that. >>>> > -- >>>> > Sylvain >>>> >>>> -- >>>> Sorry this was sent from mobile. Will do less grammar and spell check >>>> than usual. >>>> >>> >>> >> > --089e01176b1d4f480104f2db8eb3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
The only= thing you really can not do CQL3 loses some of the concept of CQL2 metadat= a, namedly the default validation and then column specific validation.

In cassandra-cql we can say (butche= ring the syntax)

create column family x
DEFAULT_VALIDATOR = =3D UTF8Type=A0
columns named y are int
columns named z = are string

You can do this in CQL.:
create table x (
rowkey blob,
column = blob,
value blob,
primary key(rowkey,column) using compact storage ;=

But you lost the concept of columns named y validate as int.= Everything is just blob as far as CQL understands it. That being said in t= he schema presented nothing stops the user from implementing their design i= n either "system"



On = Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sylvain@datastax.com> wrote:
=
On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <woo= lfel@gmail.com> wrote:

I disa= gree with the sentiment that "thrift is not worth the trouble".

Way to quote= only part of my sentence and get mental on it. My full sentence was "= it's probably not worth the trouble to start with thrift if you're = gonna use CQL later".
=A0

CQL and all SQL inspired dialects limit one's ability to use arbitrary = typed data in dynamic columns. With thrift it's easy and straight forwa= rd. With CQL there is no way to tell Cassandra the type of the name and val= ue for a dynamic column. You can only set the default type. That means usin= g a "pure cql" approach you can deviate from the default type. Ca= ssandra will throw an exception indicating the type is different than the d= efault type.=A0
=

Until such time that CQL abandons the shackles of SQL and adds th= e ability to indicate the column and value type. Something like this
<= /div>

insert into myColumnFamily(staticColum= n1, staticColumn2, 20 as int, dynamicColumn as string) into ('text1'= ;,'text2',30.55 as double, 3500 as long)

This is one area where Thrift is superior to CQL. Having said tha= t, it's valid to use Cassandra "as if" it was a relational da= tabase, but then you'd miss out on some of the unique features.

Man, if I had a nickel every time so= meone came on that mailing list pretending that something was possible with= thrift and not CQL ... I will claim this: with=A0CASSANDRA-6561 and CASSAN= DRA-4851=A0that just got in, there is *nothing* that thrift can do that CQL= cannot. But well, what do I know about Cassandra.

--
Sylvain=A0

=A0




On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sylvain@= datastax.com> wrote:
On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <edlinuxguru@gmail.co= m> wrote:
For what it is= worth you schema is simple and uses compact storage. Thus you really dont = need anything in cassandra 2.0 as far as i can tell. You might be happier w= ith a stable release like 1.2.something and just hector or astyanax. You ar= e really dealing with many issues you should not have to just to protoype a= simple cassandra app.


Of course, if everyone was using t= hat reasoning, no-one would ever test new features and report problems/sugg= est improvement. So thanks to anyone like R=FCdiger that actually tries stu= ff and take the time to report problems when they think they encounter one.= Keep at it, *you* are the one helping Cassandra to get better everyday.=A0=

And you are also right=A0R=FCdiger that it's probab= ly not worth the trouble to start with thrift if you're gonna use CQL l= ater. And you definitively should use CQL, it is Cassandra's future.

--
Sylvain=A0

<= div>


On Thursday, February 20, 2014, Sylvain Lebresne <sylvain@datastax.com> wrote:=
>
>
>
> On Wed, Feb 19, 2014 at 9:38 PM, R=FCdiger= Klaehn <rklaehn@= gmail.com> wrote:
>>
>> I have cloned the cassandra repo, applied the patch, a= nd built it. But when I want to run the bechmark I get an exception. See be= low. I tried with a non-managed dependency to cassandra-driver-core-2.0.0-r= c3-SNAPSHOT-jar-with-dependencies.jar, which I compiled from source because= I read that that might help. But that did not make a difference.
>>
>> So currently I don't know how to give the patch a = try. Any ideas?
>>
>> cheers,
>>
>> R= =FCdiger
>>
>> Exception in thread "main" java.= lang.IllegalArgumentException: replicate_on_write is not a column defined i= n this metadata
>> =A0=A0=A0 at com.datastax.driver.core.ColumnDefinitions.getAllIdx(= ColumnDefinitions.java:273)
>> =A0=A0=A0 at com.datastax.driver.co= re.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>> = =A0=A0=A0 at com.datastax.driver.core.Row.getBool(Row.java:117)
>> =A0=A0=A0 at com.datastax.driver.core.TableMetadata$Options.<in= it>(TableMetadata.java:474)
>> =A0=A0=A0 at com.datastax.driver= .core.TableMetadata.build(TableMetadata.java:107)
>> =A0=A0=A0 at = com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>> =A0=A0=A0 at com.datastax.driver.core.Metadata.rebuildSchema(Metad= ata.java:89)
>> =A0=A0=A0 at com.datastax.driver.core.ControlConne= ction.refreshSchema(ControlConnection.java:259)
>> =A0=A0=A0 at co= m.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:= 214)
>> =A0=A0=A0 at com.datastax.driver.core.ControlConnection.reconnectI= nternal(ControlConnection.java:161)
>> =A0=A0=A0 at com.datastax.d= river.core.ControlConnection.connect(ControlConnection.java:77)
>>= =A0=A0=A0 at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:89= 0)
>> =A0=A0=A0 at com.datastax.driver.core.Cluster$Manager.newSession(C= luster.java:910)
>> =A0=A0=A0 at com.datastax.driver.core.Cluster$= Manager.access$200(Cluster.java:806)
>> =A0=A0=A0 at com.datastax.= driver.core.Cluster.connect(Cluster.java:158)
>> =A0=A0=A0 at cassandra.CassandraTestMinimized$delayedInit$body.app= ly(CassandraTestMinimized.scala:31)
>> =A0=A0=A0 at scala.Function= 0$class.apply$mcV$sp(Function0.scala:40)
>> =A0=A0=A0 at scala.run= time.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>> =A0=A0=A0 at scala.App$$anonfun$main$1.apply(App.scala:71)
>= > =A0=A0=A0 at scala.App$$anonfun$main$1.apply(App.scala:71)
>>= =A0=A0=A0 at scala.collection.immutable.List.foreach(List.scala:318)
&g= t;> =A0=A0=A0 at scala.collection.generic.TraversableForwarder$class.for= each(TraversableForwarder.scala:32)
>> =A0=A0=A0 at scala.App$class.main(App.scala:71)
>> =A0=A0= =A0 at cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:= 5)
>> =A0=A0=A0 at cassandra.CassandraTestMinimized.main(Cassandra= TestMinimized.scala)
>
> I believe you've tried the cassandra trunk branch? trunk i= s basically the future Cassandra 2.1 and the driver is currently unhappy be= cause the replicate_on_write option has been removed in that version. I'= ;m supposed to have fixed that on the driver 2.0 branch like 2 days ago so = maybe you're also using a slightly old version of the driver sources in= there? Or maybe I've screwed up my fix, I'll double check. But any= way, it would be overall simpler to test with the cassandra-2.0 branch of C= assandra, with which you shouldn't run into that.
> --
> Sylvain
<= br>
--
Sorry this was sent from mobile. Will= do less grammar and spell check than usual.




--089e01176b1d4f480104f2db8eb3--