cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Sanda <john.sa...@gmail.com>
Subject Re: calculating sizes on disk
Date Sat, 07 Dec 2013 16:50:24 GMT
I finally got the math right for the partition index after tracing through
SSTableWriter.IndexWriter.append(DecoratedKey key, RowIndexEntry
indexEntry). I should also note that I am working off of the source for
1.2.9. Here is the break down for what gets written to disk in the append()
call (my keys are 4 bytes  while column names and values are both 8 bytes).

// key
key length - 2 bytes
key - 4 bytes

// index entry
index entry position - 8 bytes
index entry size - 4 bytes

// when the index entry contains columns index, the following entries will
be written ceil(total_row_size / column_index_size_in_kb) times
local deletion time - 4 bytes
marked for delete at - 8 bytes
columns index entry first name length - 2 bytes
columns index entry first name - 8 bytes
columns index entry last name length - 2 bytes
columns index entry last name - 8 bytes
columns index entry offset - 8 bytes
columns index entry width - 8 bytes

I also went through the serialization code for bloom filters, but I do not
understand the math. Even with my slightly improved understanding, I am
still uncertain about how effective any sizing analysis will be since the
numbers of rows and columns will vary per SSTable.


On Fri, Dec 6, 2013 at 3:53 PM, John Sanda <john.sanda@gmail.com> wrote:

> I have done that, but it only gets me so far because the cluster and app
> that manages it is run by 3rd parties. Ideally, I would like to provide my
> end users with a formula or heuristic for establishing some sort of
> baselines that at least gives them a general idea for planning. Generating
> data as you have suggested and as I have done is helpful, but it is hard
> for users to extrapolate out from that.
>
>
> On Fri, Dec 6, 2013 at 3:47 PM, Jacob Rhoden <jacob.rhoden@me.com> wrote:
>
>> Not sure what your end setup will be, but I would probably just spin up a
>> cluster and fill it with typical data to and measure the size on disk.
>>
>> ______________________________
>> Sent from iPhone
>>
>> On 7 Dec 2013, at 6:08 am, John Sanda <john.sanda@gmail.com> wrote:
>>
>> I am trying to do some disk capacity planning. I have been referring the
>> datastax docs[1] and this older blog post[2]. I have a column family with
>> the following,
>>
>> row key - 4 bytes
>> column name - 8 bytes
>> column value - 8 bytes
>> max number of non-deleted columns per row - 20160
>>
>> Is there an effective way to calculate the sizes (or at least a decent
>> approximation) of the bloom filters and partition indexes on disk?
>>
>> [1] Calculating user data size<http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html>
>> [2] Cassandra Storage Sizing <http://btoddb-cass-storage.blogspot.com/>
>>
>> --
>>
>> - John
>>
>>
>
>
> --
>
> - John
>



-- 

- John

Mime
View raw message