you know.. one thing I failed to mention.. .is that this is going into a "bucket" and while it's a logical row, the physical row is like 500MB … according to compaction logs.

is the ENTIRE physical row going into the cache as one unit?  That's definitely going to be a problem in this model.  500MB is a big atomic unit.

also.. I assume it's having to do a binary search within the physical row ? 


On Tue, Jul 1, 2014 at 5:54 PM, Kevin Burton <burton@spinn3r.com> wrote:
I'm really perplexed on this one..  I think this must be a bug or some misconfiguration somewhere.

I'm fetching ONE row which is one cell in my table.  

The row is in the row cache, sitting in memory.

SELECT sequence from content where bucket=98 AND sequence = 1403481494000005742;

And look at the trace below… it's taking 1000ms to go to the row cache?  That seems insane! 

Something must be broken somewhere.. Any advice in what I should be looking into?

The VM has plenty of memory so I don't think it's GC.

 activity                                                                                             | timestamp    | source      | source_elapsed
------------------------------------------------------------------------------------------------------+--------------+-------------+----------------
                                                                                   execute_cql3_query | 19:48:48,559 | 10.24.23.94 |              0
 Parsing SELECT sequence from content where bucket=98 AND sequence = 1403481494000005742 LIMIT 10000; | 19:48:48,559 | 10.24.23.94 |             63
                                                                                  Preparing statement | 19:48:48,559 | 10.24.23.94 |            153
                                                                                        Row cache hit | 19:48:49,706 | 10.24.23.94 |        1147108
                                                                   Read 1 live and 0 tombstoned cells | 19:48:49,706 | 10.24.23.94 |        1147236
                                                                                     Request complete | 19:48:49,706 | 10.24.23.94 |        1147412


--

Founder/CEO Spinn3r.com
Location: San Francisco, CA
… or check out my Google+ profile




--

Founder/CEO Spinn3r.com
Location: San Francisco, CA
blog: http://burtonator.wordpress.com
… or check out my Google+ profile