cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Ancona <>
Subject Re: Data Modeling: Partition Size and Query Efficiency
Date Wed, 06 Jan 2016 14:04:54 GMT
On Tue, Jan 5, 2016 at 5:52 PM, Jonathan Haddad <> wrote:

> You could keep a "num_buckets" value associated with the client's account,
> which can be adjusted accordingly as usage increases.

Yes, but the adjustment problem is tricky when there are multiple
concurrent writers. What happens when you change the number of buckets?
Does existing data have to be re-written into new buckets? If so, how do
you make sure that's only done once for each bucket size increase? Or
perhaps I'm misunderstanding your suggestion?


> On Tue, Jan 5, 2016 at 2:17 PM Jim Ancona <> wrote:
>> On Tue, Jan 5, 2016 at 4:56 PM, Clint Martin <
>>> wrote:
>>> What sort of data is your clustering key composed of? That might help
>>> some in determining a way to achieve what you're looking for.
>> Just a UUID that acts as an object identifier.
>>> Clint
>>> On Jan 5, 2016 2:28 PM, "Jim Ancona" <> wrote:
>>>> Hi Nate,
>>>> Yes, I've been thinking about treating customers as either small or
>>>> big, where "small" ones have a single partition and big ones have 50 (or
>>>> whatever number I need to keep sizes reasonable). There's still the problem
>>>> of how to handle a small customer who becomes too big, but that will happen
>>>> much less frequently than a customer filling a partition.
>>>> Jim
>>>> On Tue, Jan 5, 2016 at 12:21 PM, Nate McCall <>
>>>> wrote:
>>>>>> In this case, 99% of my data could fit in a single 50 MB partition.
>>>>>> But if I use the standard approach, I have to split my partitions
into 50
>>>>>> pieces to accommodate the largest data. That means that to query
the 700
>>>>>> rows for my median case, I have to read 50 partitions instead of
>>>>>> If you try to deal with this by starting a new partition when an
>>>>>> one fills up, you have a nasty distributed consensus problem, along
>>>>>> read-before-write. Cassandra LWT wasn't available the last time I
>>>>>> with this, but might help with the consensus part today. But there
>>>>>> still some nasty corner cases.
>>>>>> I have some thoughts on other ways to solve this, but they all have
>>>>>> drawbacks. So I thought I'd ask here and hope that someone has a
>>>>>> approach.
>>>>> Hi Jim - good to see you around again.
>>>>> If you can segment this upstream by customer/account/whatever,
>>>>> handling the outliers as an entirely different code path (potentially
>>>>> different cluster as the workload will be quite different at that point
>>>>> have different tuning requirements) would be your best bet. Then a
>>>>> read-before-write makes sense given it is happening on such a small number
>>>>> of API queries.
>>>>> --
>>>>> -----------------
>>>>> Nate McCall
>>>>> Austin, TX
>>>>> @zznate
>>>>> Co-Founder & Sr. Technical Consultant
>>>>> Apache Cassandra Consulting

View raw message