cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Golick <>
Subject Row slice / cache performance
Date Sun, 02 May 2010 18:00:40 GMT
   1. I wrote the list a while back about less-than-great performance when
   reading thousands of columns even on cache hits. Last night, I decided to
   try to get to the bottom of why.

I tested this by setting the row cache capacity on a TimeUUIDType-sorted CF
to 10, filling up a single row with 2000 columns, and only running queries
against that row. That row was the only thing in the database. I rm -Rf'd
the data before starting the test.

The tests were done from Coda Hale's scala client cassie, which is just a
thin layer around the java thrift bindings. I didn't actually time each call
because that wasn't the objective, but I didn't really need to. Reads of 10
columns felt quick enough, but 100 columns was slower. 1000 columns would
frequently cause the client to timeout. The cache hit rate on that CF was
1.0, so, yes, the row was in cache.

Doing a thousand reads with count=100 in a single thread pegged my macbook's
CPU and caused the fans to spin up pretty loud.

So, I attached a profiler and repeated the test. I'm no expert on cassandra
internals, so please let me know if I'm way off here. The profiled reads
were reversed=true, count=100.

As far as I can tell, there are three components taking up most of the time
on this type of read (row slice out of cache):

   1. ColumnFamilystore.removeDeleted() @ ~40% - Most of the time in here is
   actually spent materializing UUID objects so that they can be compared in
   the ConcurrentSkipListMap (ColumnFamily.columns_).
   2. SliceQueryFilter.getMemColumnIterator @ ~30% - Virtually all the time
   in here is spent in ConcurrentSkipListMap$Values.toArrray()
   3. QueryFilter.collectCollatedColumns @ ~30% - All the time being spent
   in ColumnFamily.addColumn, and about half of the total spent materializing
   UUIDs for comparison.

This profile is consistent with the decrease in performance with higher
values for count. If there are more UUIDs to deserialize, the performance of
removeDeleted(), and collectCollatedColumns() should increase (roughly)

So, my question at this point is how to fix it. I have some basic ideas, but
being new to cassandra internals, I'm not sure they make any sense. Help me
out here:

   1. Optionally call removeDeleted() less often. I realize that this is
   probably a bad idea for a lot of reasons, but it was the first thing I
   thought of.
   2. When a ColumnFamily object is put in to the row cache, copy the
   columns over to another data structure that doesn't need to be sorted on
   get(). If columns_ needs to be kept around, this option would have a memory
   impact, but at least for us, it'd be well worth it for the speed.
   3. ????

I'd love to hear feedback on these / the rest of this (long) post.

View raw message