cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryce Allen <bal...@ci.uchicago.edu>
Subject Re: two dimensional slicing
Date Thu, 26 Jan 2012 16:31:25 GMT
Thanks, comments inline:

On Mon, 23 Jan 2012 20:59:34 +1300
aaron morton <aaron@thelastpickle.com> wrote:
> It depends a bit on the data and the query patterns. 
> 
> * How many versions do you have ? 
We may have 10k versions in some cases, with up to a million names
total in any given version but more often <10K. To manage this we are
currently using two CFs, one for storing compacted complete lists and
one for storing deltas on the compacted list. Based on usage, we will
create a new compacted list and start writing deltas against that. We
should be able to limit the number of deltas in a single row to below
100; I'd like to be able to keep it lower but I'm not sure we can
maintain that under all load scenarios. The compacted lists are
straightforward, but there are many ways to structure the deltas and
they all have trade offs. A CF with composite columns that supported
two dimensional slicing would be perfect.

> * How many names in each version ?
We plan on limiting to a total of 1 million names, and around 10,000 per
version (by limiting the batch size), but many deltas will have <10
names.

> * When querying do you know the versions numbers you want to query
> from ? How many are there normally?
Currently we don't know the version numbers in advance - they are
timestamps, and we are querying for versions less than or equal to the
desired timestamp. We have talked about using vector clock versions and
maintaining an index mapping time to version numbers, in which case we
would know the exact versions after the index lookup, at the expense of
another RTT on every operation.

> * How frequent are the updates and the reads ?
We expect reads to be more frequent than writes. Unfortunately we don't
have solid numbers on what to expect, but I would guess 20x. Update
operations will involve several reads to determine where to write.


> I would lean towards using two standard CF's, one to list all the
> version numbers (in a single row probably) and one to hold the names
> in a particular version. 
> 
> To do your query slice the first CF and then run multi gets to the
> second. 
> 
> Thats probably not the best solution, if you can add some more info
> it may get better.
I'm actually leaning back toward BOP, as I run into more issues
and complexity with the RP models. I'd really like to implement both
and compare them, but at this point I need to focus on one to get
things working, so I'm trying to make a best initial guess.


> 
> On 21/01/2012, at 6:20 AM, Bryce Allen wrote:
> 
> > I'm storing very large versioned lists of names, and I'd like to
> > query a range of names within a given range of versions, which is a
> > two dimensional slice, in a single query. This is easy to do using
> > ByteOrderedPartitioner, but seems to require multiple (non parallel)
> > queries and extra CFs when using RandomPartitioner.
> > 
> > I see two approaches when using RP:
> > 
> > 1) Data is stored in a super column family, with one dimension being
> > the super column names and the other the sub column names. Since
> > slicing on sub columns requires a list of super column names, a
> > second standard CF is needed to get a range of names before doing a
> > query on the main super CF. With CASSANDRA-2710, the same is
> > possible using a standard CF with composite types instead of a
> > super CF.
> > 
> > 2) If one of the dimensions is small, a two dimensional slice isn't
> > required. The data can be stored in a standard CF with linear
> > ordering on a composite type (large_dimension, small_dimension).
> > Data is queried based on the large dimension, and the client throws
> > out the extra data in the other dimension.
> > 
> > Neither of the above solutions are ideal. Does anyone else have a
> > use case where two dimensional slicing is useful? Given the
> > disadvantages of BOP, is it practical to make the composite column
> > query model richer to support this sort of use case?
> > 
> > Thanks,
> > Bryce
> 

Mime
View raw message