I turned on debug logging for a few days and timeouts happened across pretty much all requests. I couldn't see any particular request that was consistently the problem.

After some experimenting it seems that shutting down cassandra and restarting resolves the problem. Once it hits the JVM memory limit however, the timeouts start again. I have read the page on MemTable thresholds and have tried thresholds of 32MB, 64MB and 128MB with no noticeable difference. Cassandra is set to use 7GB of memory. I have 12 CF's, however only 6 of those have lots of data.

Cheers,
Chris

On Tue, Nov 10, 2009 at 11:55 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
if you're timing out doing a slice on 10 columns w/ 10% cpu used,
something is broken

is it consistent as to which keys this happens on?  try turning on
debug logging and seeing where the latency is coming from.

On Tue, Nov 10, 2009 at 1:53 PM, Chris Were <chris.were@gmail.com> wrote:
>
> On Tue, Nov 10, 2009 at 11:50 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> On Tue, Nov 10, 2009 at 1:49 PM, Chris Were <chris.were@gmail.com> wrote:
>> > Maybe... but it's not just multigets, it also happens when retreiving
>> > one
>> > row with get_slice.
>>
>> how many of the 3M columns are you trying to slice at once?
>
> Sorry, I must have mixed up the terminology.
> There's ~3M keys, but less than 10 columns in each. The get_slice calls are
> to retreive all the columns (10) for a given key.