incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryce Allen <bal...@ci.uchicago.edu>
Subject Re: Choosing a Partitioner Type for Random java.util.UUID Row Keys
Date Thu, 22 Dec 2011 17:29:03 GMT
Thanks, that definitely has advantages over using a super column. We
ran into thrift timeouts when the super column got large, and with the
super column range query there is no way (AFAIK) to batch the request at
the subcolumn level.

-Bryce

On Thu, 22 Dec 2011 10:06:58 +1300
aaron morton <aaron@thelastpickle.com> wrote:
> AFAIK there are no plans kill the BOP, but I would still try to make
> your life easier by using the RP. . 
> 
> My understanding of the problem is at certain times you snapshot the
> files in a dir; and the main query you want to handle is "At what
> points between time t0 and time t1 did files x,y and z exist?".
> 
> You could consider:
> 
> 1) Partitioning the time series data in across each row, then make
> the row key is the timestamp for the start of the partition. If you
> have rollup partitions consider making the row key <timestamp :
> partition_size> , e.g. <123456789."1d"> for a 1 day partition that
> starts at 123456789 2) In each row use column names that have the
> form <timestamp : file_name> where time stamp is the time of the
> snapshot. 
> 
> To query between two times (t0 and t1):
> 
> 1) Determine which partitions the time span covers, this will give
> you a list of rows. 2) Execute a multi-get slice for the all rows
> using  <t0:*> and <t1:*> (I'm using * here as a null, check with your
> client to see how to use composite columns.)
> 
> Hope that helps. 
> Aaron
> 
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 21/12/2011, at 9:03 AM, Bryce Allen wrote:
> 
> > I wasn't aware of CompositeColumns, thanks for the tip. However I
> > think it still doesn't allow me to do the query I need - basically
> > I need to do a timestamp range query, limiting only to certain file
> > names at each timestamp. With BOP and a separate row for each
> > timestamp, prefixed by a random UUID, and file names as column
> > names, I can do this query. With CompositeColumns, I can only query
> > one contiguous range, so I'd have to know the timestamps before
> > hand to limit the file names. I can resolve this using indexes, but
> > on paper it looks like this would be significantly slower (it would
> > take me 5 round trips instead of 3 to complete each query, and the
> > query is made multiple times on every single client request).
> > 
> > The two down sides I've seen listed for BOP are balancing issues and
> > hotspots. I can understand why RP is recommended, from the balancing
> > issues alone. However these aren't problems for my application. Is
> > there anything else I am missing? Does the Cassandra team plan on
> > continuing to support BOP? I haven't completely ruled out RP, but I
> > like having BOP as an option, it opens up interesting modeling
> > alternatives that I think have real advantages for some
> > (if uncommon) applications.
> > 
> > Thanks,
> > Bryce
> > 
> > On Wed, 21 Dec 2011 08:08:16 +1300
> > aaron morton <aaron@thelastpickle.com> wrote:
> >> Bryce, 
> >> 	Have you considered using CompositeColumns and a standard
> >> CF? Row key is the UUID column name is (timestamp : dir_entry) you
> >> can then slice all columns with a particular time stamp. 
> >> 
> >> 	Even if you have a random key, I would use the RP unless
> >> you have an extreme use case. 
> >> 
> >> Cheers
> >> 
> >> -----------------
> >> Aaron Morton
> >> Freelance Developer
> >> @aaronmorton
> >> http://www.thelastpickle.com
> >> 
> >> On 21/12/2011, at 3:06 AM, Bryce Allen wrote:
> >> 
> >>> I think it comes down to how much you benefit from row range
> >>> scans, and how confident you are that going forward all data will
> >>> continue to use random row keys.
> >>> 
> >>> I'm considering using BOP as a way of working around the non
> >>> indexes super column limitation. In my current schema, row keys
> >>> are random UUIDs, super column names are timestamps, and columns
> >>> contain a snapshot in time of directory contents, and could be
> >>> quite large. If instead I use row keys that are
> >>> (uuid)-(timestamp), and use a standard column family, I can do a
> >>> row range query and select only specific columns. I'm still
> >>> evaluating if I can do this with BOP - ideally the token would
> >>> just use the first 128 bits of the key, and I haven't found any
> >>> documentation on how it compares keys of different length.
> >>> 
> >>> Another trick with BOP is to use MD5(rowkey)-rowkey for data that
> >>> has non uniform row keys. I think it's reasonable to use if most
> >>> data is uniform and benefits from range scans, but a few things
> >>> are added that aren't/don't. This trick does make the keys larger,
> >>> which increases storage cost and IO load, so it's probably a bad
> >>> idea if a significant subset of the data requires it.
> >>> 
> >>> Disclaimer - I wrote that wiki article to fill in a documentation
> >>> gap, since there were no examples of BOP and I wasted a lot of
> >>> time before I noticed the hex byte array vs decimal distinction
> >>> for specifying the initial tokens (which to be fair is
> >>> documented, just easy to miss on a skim). I'm also new to
> >>> cassandra, I'm just describing what makes sense to me "on paper".
> >>> FWIW I confirmed that random UUIDs (type 4) row keys really do
> >>> evenly distribute when using BOP.
> >>> 
> >>> -Bryce
> >>> 
> >>> On Mon, 19 Dec 2011 19:01:00 -0800
> >>> Drew Kutcharian <drew@venarc.com> wrote:
> >>>> Hey Guys,
> >>>> 
> >>>> I just came across
> >>>> http://wiki.apache.org/cassandra/ByteOrderedPartitioner and it
> >>>> got me thinking. If the row keys are java.util.UUID which are
> >>>> generated randomly (and securely), then what type of partitioner
> >>>> would be the best? Since the key values are already random,
> >>>> would it make a difference to use RandomPartitioner or one can
> >>>> use ByteOrderedPartitioner or OrderPreservingPartitioning as
> >>>> well and get the same result?
> >>>> 
> >>>> -- Drew
> >>>> 
> >> 
> 

Mime
View raw message