incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: How does cfstats calculate Row Size?
Date Fri, 13 Aug 2010 04:11:57 GMT
Right, row stats in 0.6 are just "what I've seen during the
compactions that happened to run since this node restarted last."

0.7 has persistent (and more fine-grained) statistics.

On Thu, Aug 12, 2010 at 1:28 PM, Ryan King <ryan@twitter.com> wrote:
> On Thu, Aug 12, 2010 at 9:08 AM, Julie <julie.sugar@nextcentury.com> wrote:
>> I am chasing down a row size discrepancy and am confused.
>>
>> I populated a single node Cassandra cluster with 10,000 rows of data, using
>> numeric keys 1-10,000, where each row is a little over 100kB in length and has
>> a single column in it.
>>
>> When I perform a cfstats on the node immediately after writing the data, it
>> reports that the Compacted row minimum size = Compacted row maximum size which
>> is a little over 100,000 bytes.  This is what I expect.
>>
>> I then run an application that randomly reads rows and adds a timestamp column
>> to each row read.  This timestamp column name and column value is just adding
>> a few bytes to the row.
>>
>> But after running my reading app for a few hours, cfstats reports a very odd
>> minimum row size (and compacted mean row size):
>>
>> [root@ec2-server1 ~]# /mnt/server/apache-cassandra-0.6.2/bin/nodetool -h
>> ec2-server1 -p 8080 cfstats
>> Keyspace: Keyspace1
>>        Read Count: 670434
>>        Read Latency: 36.22349047035205 ms.
>>        Write Count: 1519933
>>        Write Latency: 0.02940705741634664 ms.
>>        Pending Tasks: 0
>>                Column Family: Standard1
>>                SSTable count: 6
>>                Space used (live): 11130225642
>>                Space used (total): 11130225642
>>                Memtable Columns Count: 1435
>>                Memtable Data Size: 40344907
>>                Memtable Switch Count: 1329
>>                Read Count: 670434
>>                Read Latency: 41.768 ms.
>>                Write Count: 1519933
>>                Write Latency: 0.025 ms.
>>                Pending Tasks: 0
>>                Key cache capacity: 200000
>>                Key cache size: 200000
>>                Key cache hit rate: 0.48049934471509675
>>                Row cache: disabled
>>                Compacted row minimum size: 238
>>                Compacted row maximum size: 100323
>>                Compacted row mean size: 67548
>>
>> I thought I had a bug in my code so I wrote another app to read every row
>> in the database, keys 1-10,000.  I get the size of each row after reading it
>> (by adding up all column names and column values in the row and the size of
>> the key string) and this matches what I expect -- every single key in this
>> table has a size of just over 100,000 bytes.  (I know there are some
>> overhead columns in each row but I assume these will only make the row
>> larger, not smaller.)
>>
>> So I am confused about where cfstats is getting the row sizes it is working
>> with?
>>
>> When I add the timestamp column to each row, I am not deleting the other
>> column (large) in the row but I am not rewriting the large column either.
>
> I'm guessing (haven't read this part of the source) that the min size
> is being generated in minor compaction, which doesn't see the whole
> row.
>
> -ryan
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message