incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Choosing a Partitioner Type for Random java.util.UUID Row Keys
Date Fri, 23 Dec 2011 09:56:11 GMT
No problems. 

IMHO you should develop a sizable bruise banging your head against a using Standard CF's and
the Random Partitioner before using something else. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 23/12/2011, at 6:29 AM, Bryce Allen wrote:

> Thanks, that definitely has advantages over using a super column. We
> ran into thrift timeouts when the super column got large, and with the
> super column range query there is no way (AFAIK) to batch the request at
> the subcolumn level.
> 
> -Bryce
> 
> On Thu, 22 Dec 2011 10:06:58 +1300
> aaron morton <aaron@thelastpickle.com> wrote:
>> AFAIK there are no plans kill the BOP, but I would still try to make
>> your life easier by using the RP. . 
>> 
>> My understanding of the problem is at certain times you snapshot the
>> files in a dir; and the main query you want to handle is "At what
>> points between time t0 and time t1 did files x,y and z exist?".
>> 
>> You could consider:
>> 
>> 1) Partitioning the time series data in across each row, then make
>> the row key is the timestamp for the start of the partition. If you
>> have rollup partitions consider making the row key <timestamp :
>> partition_size> , e.g. <123456789."1d"> for a 1 day partition that
>> starts at 123456789 2) In each row use column names that have the
>> form <timestamp : file_name> where time stamp is the time of the
>> snapshot. 
>> 
>> To query between two times (t0 and t1):
>> 
>> 1) Determine which partitions the time span covers, this will give
>> you a list of rows. 2) Execute a multi-get slice for the all rows
>> using  <t0:*> and <t1:*> (I'm using * here as a null, check with your
>> client to see how to use composite columns.)
>> 
>> Hope that helps. 
>> Aaron
>> 
>> 
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 21/12/2011, at 9:03 AM, Bryce Allen wrote:
>> 
>>> I wasn't aware of CompositeColumns, thanks for the tip. However I
>>> think it still doesn't allow me to do the query I need - basically
>>> I need to do a timestamp range query, limiting only to certain file
>>> names at each timestamp. With BOP and a separate row for each
>>> timestamp, prefixed by a random UUID, and file names as column
>>> names, I can do this query. With CompositeColumns, I can only query
>>> one contiguous range, so I'd have to know the timestamps before
>>> hand to limit the file names. I can resolve this using indexes, but
>>> on paper it looks like this would be significantly slower (it would
>>> take me 5 round trips instead of 3 to complete each query, and the
>>> query is made multiple times on every single client request).
>>> 
>>> The two down sides I've seen listed for BOP are balancing issues and
>>> hotspots. I can understand why RP is recommended, from the balancing
>>> issues alone. However these aren't problems for my application. Is
>>> there anything else I am missing? Does the Cassandra team plan on
>>> continuing to support BOP? I haven't completely ruled out RP, but I
>>> like having BOP as an option, it opens up interesting modeling
>>> alternatives that I think have real advantages for some
>>> (if uncommon) applications.
>>> 
>>> Thanks,
>>> Bryce
>>> 
>>> On Wed, 21 Dec 2011 08:08:16 +1300
>>> aaron morton <aaron@thelastpickle.com> wrote:
>>>> Bryce, 
>>>> 	Have you considered using CompositeColumns and a standard
>>>> CF? Row key is the UUID column name is (timestamp : dir_entry) you
>>>> can then slice all columns with a particular time stamp. 
>>>> 
>>>> 	Even if you have a random key, I would use the RP unless
>>>> you have an extreme use case. 
>>>> 
>>>> Cheers
>>>> 
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>> 
>>>> On 21/12/2011, at 3:06 AM, Bryce Allen wrote:
>>>> 
>>>>> I think it comes down to how much you benefit from row range
>>>>> scans, and how confident you are that going forward all data will
>>>>> continue to use random row keys.
>>>>> 
>>>>> I'm considering using BOP as a way of working around the non
>>>>> indexes super column limitation. In my current schema, row keys
>>>>> are random UUIDs, super column names are timestamps, and columns
>>>>> contain a snapshot in time of directory contents, and could be
>>>>> quite large. If instead I use row keys that are
>>>>> (uuid)-(timestamp), and use a standard column family, I can do a
>>>>> row range query and select only specific columns. I'm still
>>>>> evaluating if I can do this with BOP - ideally the token would
>>>>> just use the first 128 bits of the key, and I haven't found any
>>>>> documentation on how it compares keys of different length.
>>>>> 
>>>>> Another trick with BOP is to use MD5(rowkey)-rowkey for data that
>>>>> has non uniform row keys. I think it's reasonable to use if most
>>>>> data is uniform and benefits from range scans, but a few things
>>>>> are added that aren't/don't. This trick does make the keys larger,
>>>>> which increases storage cost and IO load, so it's probably a bad
>>>>> idea if a significant subset of the data requires it.
>>>>> 
>>>>> Disclaimer - I wrote that wiki article to fill in a documentation
>>>>> gap, since there were no examples of BOP and I wasted a lot of
>>>>> time before I noticed the hex byte array vs decimal distinction
>>>>> for specifying the initial tokens (which to be fair is
>>>>> documented, just easy to miss on a skim). I'm also new to
>>>>> cassandra, I'm just describing what makes sense to me "on paper".
>>>>> FWIW I confirmed that random UUIDs (type 4) row keys really do
>>>>> evenly distribute when using BOP.
>>>>> 
>>>>> -Bryce
>>>>> 
>>>>> On Mon, 19 Dec 2011 19:01:00 -0800
>>>>> Drew Kutcharian <drew@venarc.com> wrote:
>>>>>> Hey Guys,
>>>>>> 
>>>>>> I just came across
>>>>>> http://wiki.apache.org/cassandra/ByteOrderedPartitioner and it
>>>>>> got me thinking. If the row keys are java.util.UUID which are
>>>>>> generated randomly (and securely), then what type of partitioner
>>>>>> would be the best? Since the key values are already random,
>>>>>> would it make a difference to use RandomPartitioner or one can
>>>>>> use ByteOrderedPartitioner or OrderPreservingPartitioning as
>>>>>> well and get the same result?
>>>>>> 
>>>>>> -- Drew
>>>>>> 
>>>> 
>> 


Mime
View raw message