incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Document storage
Date Thu, 29 Mar 2012 19:35:52 GMT
I kind of hijacked
https://issues.apache.org/jira/browse/CASSANDRA-3647 ("Sylvain
suggests we start with (non-nested) lists, maps, and sets. I agree
that this is a great 80/20 approach to the problem") but we could
split it out to another ticket.

On Thu, Mar 29, 2012 at 2:24 PM, Ben McCann <ben@benmccann.com> wrote:
> Thanks Jonathan.  The only reason I suggested JSON was because it already
> has support for lists.  Native support for lists in Cassandra would more
> than satisfy me.  Are there any existing proposals or a bug I can follow?
>  I'm not familiar with the Cassandra codebase, so I'm not entirely sure how
> helpful I can be, but I'd certainly be interested in taking a look to see
> what's required.
>
> -Ben
>
>
> On Thu, Mar 29, 2012 at 12:19 PM, Brian O'Neill <bone@alumni.brown.edu>wrote:
>
>> Jonathan,
>>
>> I was actually going to take this up with Nate McCall a few weeks back.  I
>> think it might make sense to get the client development community together
>> (Netflix w/ Astyanax, Hector, Pycassa, Virgil, etc.)
>>
>> I agree whole-heartedly that it shouldn't go into the database for all the
>> reasons you point out.
>>
>> If we can all decide on some standards for data storage (e.g. composite
>> types), indexing strategies, etc.  We can provide higher-level functions
>> through the client libraries and also provide interoperability between
>> them.  (without bloating Cassandra)
>>
>> CCing Nate.  Nate, thoughts?
>> I wouldn't mind coordinating/facilitating the conversation.  If we know
>> who should be involved.
>>
>> -brian
>>
>> ----
>> Brian O'Neill
>> Lead Architect, Software Development
>> Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
>> p: 215.588.6024blog: http://weblogs.java.net/blog/boneill42/
>> blog: http://brianoneill.blogspot.com/
>>
>>
>>
>>
>>
>>
>>
>> On 3/29/12 3:06 PM, "Ben McCann" <ben@benmccann.com> wrote:
>>
>> >Jonathan, I asked Brian about his REST
>> >API<
>> https://groups.google.com/forum/?fromgroups#!topic/virgil-users/oncBas
>> >9C8Us>and
>> >he said he does not take the json objects and split them because the
>> >client libraries do not agree on implementations.  This was exactly my
>> >concern as well with this solution.  I would be perfectly happy to do it
>> >this way instead of using JSON if it were standardized.  The reason I
>> >suggested JSON is that it is standardized.  As far as I can tell,
>> >Cassandra
>> >doesn't support maps and lists in a standardized way today, which is the
>> >root of my problem.
>> >
>> >-Ben
>> >
>> >
>> >On Thu, Mar 29, 2012 at 11:30 AM, Drew Kutcharian <drew@venarc.com>
>> wrote:
>> >
>> >> Yes, I meant the "row header index". What I have done is that I'm
>> >>storing
>> >> an object (i.e. UserProfile) where you read or write it as a whole (a
>> >>user
>> >> updates their user details in a single page in the UI). So I serialize
>> >>that
>> >> object into a binary JSON using SMILE format. I then compress it using
>> >> Snappy on the client side. So as far as Cassandra cares it's storing a
>> >> byte[].
>> >>
>> >> Now on the client side, I'm using cassandra-cli with a custom type that
>> >> knows how to turn a byte[] into a JSON text and back. The only issue was
>> >> CASSANDRA-4081 where "assume" doesn't work with custom types. If
>> >> CASSANDRA-4081 gets fixed, I'll get the best of both worlds.
>> >>
>> >> Also advantages of this vs. the thrift based Super Column families are:
>> >>
>> >> 1. Saving extra CPU usage on the Cassandra nodes. Since
>> >> serialize/deserialize and compression/decompression happens on the
>> >>client
>> >> nodes where there is plenty idle CPU time
>> >>
>> >> 2. Saving network bandwidth since I'm sending over a compressed byte[]
>> >>
>> >>
>> >> -- Drew
>> >>
>> >>
>> >>
>> >> On Mar 29, 2012, at 11:16 AM, Jonathan Ellis wrote:
>> >>
>> >> > On Thu, Mar 29, 2012 at 1:11 PM, Drew Kutcharian <drew@venarc.com>
>> >> wrote:
>> >> >>> I think this is a much better approach because that gives you
the
>> >> >>> ability to update or retrieve just parts of objects efficiently,
>> >> >>> rather than making column values just blobs with a bunch of
special
>> >> >>> case logic to introspect them.  Which feels like a big step
>> >>backwards
>> >> >>> to me.
>> >> >>
>> >> >> Unless your access pattern involves reading/writing the whole
>> >>document
>> >> each time. In that case you're better off serializing the whole document
>> >> and storing it in a column as a byte[] without incurring the overhead of
>> >> column indexes. Right?
>> >> >
>> >> > Hmm, not sure what you're thinking of there.
>> >> >
>> >> > If you mean the "index" that's part of the row header for random
>> >> > access within a row, then no, serializing to byte[] doesn't save you
>> >> > anything.
>> >> >
>> >> > If you mean secondary indexes, don't declare any if you don't want
>> >>any.
>> >> :)
>> >> >
>> >> > Just telling C* to store a byte[] *will* be slightly lighter-weight
>> >> > than giving it named columns, but we're talking negligible compared
to
>> >> > the overhead of actually moving the data on or off disk in the first
>> >> > place.  Not even close to being worth giving up being able to deal
>> >> > with your data from standard tools like cqlsh, IMO.
>> >> >
>> >> > --
>> >> > Jonathan Ellis
>> >> > Project Chair, Apache Cassandra
>> >> > co-founder of DataStax, the source for professional Cassandra support
>> >> > http://www.datastax.com
>> >>
>> >>
>>
>>
>>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message