Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 99690 invoked from network); 3 Feb 2011 15:19:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 3 Feb 2011 15:19:07 -0000 Received: (qmail 74847 invoked by uid 500); 3 Feb 2011 15:19:05 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 74601 invoked by uid 500); 3 Feb 2011 15:19:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74592 invoked by uid 99); 3 Feb 2011 15:19:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Feb 2011 15:19:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Feb 2011 15:18:54 +0000 Received: by iym1 with SMTP id 1so676962iym.31 for ; Thu, 03 Feb 2011 07:18:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.231.14.69 with SMTP id f5mr11537242iba.145.1296746312312; Thu, 03 Feb 2011 07:18:32 -0800 (PST) Received: by 10.231.15.72 with HTTP; Thu, 3 Feb 2011 07:18:32 -0800 (PST) X-Originating-IP: [80.179.102.198] In-Reply-To: References: Date: Thu, 3 Feb 2011 17:18:32 +0200 Message-ID: Subject: Re: Do supercolumns have a purpose? From: David Boxenhorn To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0003255760f64f27b2049b62471e X-Virus-Checked: Checked by ClamAV on apache.org --0003255760f64f27b2049b62471e Content-Type: text/plain; charset=ISO-8859-1 Well, I am an "actual active developer" and I have "managed to do pretty nice stuffs with Cassandra" - without secondary indexes so far. But I'm looking forward to having secondary indexes in my arsenal when new functional requirements come up, and I'm bummed out that my early design decision to use supercolums wherever I could, instead of concatenating keys, has closed off a whole lot of possibilities. I knew when I started that secondary keys were in the future, if I had known that they would be only for regular column families I wouldn't have used supercolumn families in the first place, now I'm pretty much stuck (too late to go back - we're launching in March). On Thu, Feb 3, 2011 at 4:44 PM, Sylvain Lebresne wrote: > On Thu, Feb 3, 2011 at 3:00 PM, David Boxenhorn wrote: > >> The advantage would be to enable secondary indexes on supercolumn >> families. >> > > Then I suggest opening a ticket for adding secondary indexes to supercolumn > families and voting on it. This will be 1 or 2 order of magnitude less work > than getting rid of super column internally, and probably a much better > solution anyway. > > >> I understand from this thread that indexes are supercolumn families are >> not going to be: >> >> http://www.mail-archive.com/user@cassandra.apache.org/msg09527.html >> > > I should maybe let Jonathan answer this one, but the way I understand it is > that adding secondary indexes to super column is not a top priority to > actual active developers. Not that it will never ever happen. And voting for > tickets in JIRA is one way to help make it raise its priority. > > In any case, if the goal you're pursuing is adding secondary indexes to > super column, then that's the ticket you should open, and if after careful > consideration it is decided that getting rid of super column is the best way > to reach that goal then so be it (spoiler: it is not). > > >> Which, it seems to me, effectively deprecates supercolumn families. (I >> don't see any of the three problems you brought up as overcoming this >> problem, except, perhaps, for special cases.) >> > > You're untitled to your opinions obviously but I doubt everyone share that > feeling (I don't for instance). Before 0.7, there was no secondary indexes > at all and still a bunch of people managed to do pretty nice stuffs with > Cassandra. In particular denormalized views are sometimes (often?) > preferable to secondary indexes for performance reasons. For that super > columns are quite handy. > > -- > Sylvain > > >> >> >> On Thu, Feb 3, 2011 at 3:32 PM, Sylvain Lebresne wrote: >> >>> On Thu, Feb 3, 2011 at 1:33 PM, David Boxenhorn wrote: >>> >>>> Thanks Sylvain! >>>> >>>> Can I vote for internally implementing supercolumn families as regular >>>> column families? (With a smooth upgrade process that doesn't require >>>> shutting down a live cluster.) >>>> >>> >>> I forgot to add that I don't know if this make a lot of sense. That would >>> be a fairly major refactor (so error prone), you'd still have to deal with >>> the point I mentioned in my previous mail (for range deletes you would have >>> to change the on-disk format for instance), and all this for no actual >>> benefits, even downsides actually (encoded supercolumn will take more space >>> on-disk (and on-memory)). Super columns are there and work fairly well, so >>> what would be the point ? >>> >>> I'm only just saying that 'in theory', super columns are not the super >>> shiny magical feature that give you stuff you can't hope to have with only >>> regular column family. That doesn't make then at least nice. >>> >>> That being said, you are free to create whatever ticket you want and vote >>> for it. Don't expect too much support tough :) >>> >>> >>>> What if supercolumn families were supported as regular column families + >>>> an index (on what used to be supercolumn keys)? Would that solve some >>>> problems? >>>> >>> >>> You'd still have to remember for each CF if it has this index on what >>> used to be supercolumn keys and handle those differently. Really not >>> convince this would make the code cleaner that how it is now. And making the >>> code cleaner is really the only reason I can thing of for wanting to get rid >>> of super columns internally, so ... >>> >>> >>>> >>>> >>>> On Thu, Feb 3, 2011 at 2:00 PM, Sylvain Lebresne wrote: >>>> >>>>> > Is there any advantage to using supercolumns >>>>> > (columnFamilyName[superColumnName[columnName[val]]]) instead of >>>>> regular >>>>> > columns with concatenated keys >>>>> > (columnFamilyName[superColumnName@columnName[val]])? >>>>> > >>>>> > When I designed my data model, I used supercolumns wherever I needed >>>>> two >>>>> > levels of key depth - just because they were there, and I figured >>>>> that they >>>>> > must be there for a reason. >>>>> > >>>>> > Now I see that in 0.7 secondary indexes don't work on supercolumns or >>>>> > subcolumns (is that right?), which seems to me like a very serious >>>>> > limitation of supercolumn families. >>>>> > >>>>> > It raises the question: Is there anything that supercolumn families >>>>> are good >>>>> > for? >>>>> >>>>> There is a bunch of queries that you cannot do (or less conveniently) >>>>> if you >>>>> encode super columns using regular columns with concatenated keys: >>>>> >>>>> 1) If you use regular columns with concatenated keys, the count >>>>> argument >>>>> count simple columns. With super columns it counts super columns. It >>>>> means >>>>> that you can't do "give me the 10 first super columns of this row". >>>>> >>>>> 2) If you need to get x super columns by name, you'll have to issue x >>>>> get_slice query (one of each super column). On the client side it >>>>> sucks. >>>>> Internally in Cassandra we could do it reasonably well though. >>>>> >>>>> 3) You cannot remove entire super columns since there is no support for >>>>> range >>>>> deletions. >>>>> >>>>> Moreover, the encoding with concatenated keys uses more disk space (and >>>>> less >>>>> disk used for the same information means less things to read so it may >>>>> have >>>>> a slight impact on read performance too -- it's probably really slight >>>>> on most >>>>> usage but nevertheless). >>>>> >>>>> > And here's a related question: Why can't Cassandra implement >>>>> supercolumn >>>>> > families as regular column families, internally, and give you that >>>>> > functionality? >>>>> >>>>> For the 1) and 2) above, we could deal with those internally fairly >>>>> easily I >>>>> think and rather well (which means it wouldn't be much worse >>>>> performance-wise >>>>> than with the actual implementaion of super columns, not that it would >>>>> be >>>>> better). For 3), range deletes are harder and would require more >>>>> significant >>>>> changes (that doesn't mean that Cassandra will never have it). Even >>>>> without >>>>> that, there would be the disk space lost. >>>>> >>>>> -- >>>>> Sylvain >>>>> >>>>> >>>> >>> >> > --0003255760f64f27b2049b62471e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Well, I am an "actual active developer" and I ha= ve "managed to do pretty nice stuffs with Cassandra" - without se= condary indexes so far. But I'm looking forward to having secondary ind= exes in my arsenal when new functional requirements come up, and I'm bu= mmed out that my early design decision to use supercolums wherever I could,= instead of concatenating keys, has closed off a whole lot of possibilities= . I knew when I started that secondary keys were in the future, if I had kn= own that they would be only for regular column families I wouldn't have= used supercolumn families in the first place, now I'm pretty much stuc= k (too late to go back - we're launching in March).


On Thu, Feb 3, 2011 at 4:44 PM, Sylvain = Lebresne <sylv= ain@datastax.com> wrote:
On Thu, Feb 3, 2011 at 3:00 PM, David Boxenhorn <david@l= ookin2.com> wrote:
The advantage would be to enable secondary indexes on supe= rcolumn families.

Then I su= ggest opening a ticket for adding secondary indexes to supercolumn families= and voting on it. This will be 1 or 2 order of magnitude less work than ge= tting rid of super column internally, and probably a much better solution a= nyway.
=A0
I understand from this thread that indexes are supercolumn fa= milies are not going to be:

http://www.mail-archive.com/user@cassandra.apach= e.org/msg09527.html

I s= hould maybe let Jonathan answer this one, but the way I understand it is th= at adding secondary indexes to super column is not a top priority to actual= active developers. Not that it will never ever happen. And voting for tick= ets in JIRA is one way to help make it raise its priority.

In any case, if the goal you're pursuing is adding = secondary indexes to super column, then that's the ticket you should op= en, and if after careful consideration it is decided that getting rid of su= per column is the best way to reach that goal then so be it (spoiler: it is= not).
=A0
Which, it seems to me, effectively deprecates supercolumn fam= ilies. (I don't see any of the three problems you brought up as overcom= ing this problem, except, perhaps, for special cases.)

You're untitled to your opinions= obviously but I doubt everyone share that feeling (I don't for instanc= e). Before 0.7, there was no secondary indexes at all and still a bunch of = people managed to do pretty nice stuffs with Cassandra. In particular denor= malized views are sometimes (often?) preferable to secondary indexes for pe= rformance reasons. For that super columns are quite handy.
=A0
--
Sylvain




On Thu, Feb 3, 2011 at 3:32 PM, Sylvain Lebresne <sylvain@datastax.com<= /a>> wrote:
I forgot to add that I don't kno= w if this make a lot of sense. That would be a fairly major refactor (so er= ror prone), you'd still have to deal with the point I mentioned in my p= revious mail (for range deletes you would have to change the on-disk format= for instance), and all this for no actual benefits, even downsides actuall= y (encoded supercolumn will take more space on-disk (and on-memory)).=A0Sup= er columns are there and work fairly well, so what would be the point ?

I'm only just saying that 'in theory', supe= r columns are not the super shiny magical feature that give you stuff you c= an't hope to have with only regular column family. That doesn't mak= e then at least nice.

That being said, you are free to create whatever ticket= you want and vote for it. Don't expect too much support tough :)
=
=A0
What if supercolumn families were supported as regular col= umn families + an index (on what used to be supercolumn keys)? Would that s= olve some problems?

=A0


On Thu, Feb 3, 2011 at 2:00 PM, Sylvain Lebr= esne <sylvain@datastax.com> wrote:
> Is there any advantage to using supercolumns
> = (columnFamilyName[superColumnName[columnName[val]]]) instead of regular
> columns with concatenated keys
> (columnFamilyName= [superColumnName@columnName[val]])?
>
> When I designed my data model, I used supercolumns= wherever I needed two
> levels of key depth - just because th= ey were there, and I figured that they
> must be there for a r= eason.
>
> Now I see that in 0.7 secondary indexes don't = work on supercolumns or
> subcolumns (is that right?), which s= eems to me like a very serious
> limitation of supercolumn fam= ilies.
>
> It raises the question: Is there anything that sup= ercolumn families are good
> for?

There is a bunch of queries that you cannot do (or less conveniently) if= you
encode super columns using regular columns with concatenated keys:

1) If you use regular columns with concatenated keys,= the count argument
count simple columns. With super columns it c= ounts super columns. It means
that you can't do "give me the 10 first super columns of this= row".

2) If you need to get x super columns = by name, you'll have to issue x
get_slice query (one of each = super column). On the client side it sucks.
Internally in Cassandra we could do it reasonably well though.

3) You cannot remove entire super columns since there is = no support for range
deletions.

Moreover= , the encoding with concatenated keys uses more disk space (and less
disk used for the same information means less things to read so it may= have
a slight impact on read performance too -- it's probabl= y really slight on most
usage but nevertheless).

> And here's a related question: Why can't Cassandra = implement supercolumn
> families as regular column families, i= nternally, and give you that
> functionality?=A0

For the 1) and 2) above, we could deal with those internal= ly fairly easily I
think and rather well (which means it wouldn&#= 39;t be much worse performance-wise
than with the actual implemen= taion of super columns, not that it would be
better). For 3), range deletes are harder and would require more signi= ficant
changes (that doesn't mean that Cassandra will never h= ave it). Even without
that, there would be the disk space lost.

--
Sylvain






--0003255760f64f27b2049b62471e--