cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Bridges <sean.brid...@gmail.com>
Subject Re: 10 minute cassandra pause
Date Wed, 23 Jun 2010 23:05:36 GMT
I see about 3000 lines of,

INFO [COMMIT-LOG-WRITER] 2010-06-23 16:40:29,107 CommitLog.java (line
412) Discarding obsolete commit
log:CommitLogSegment(/data1/cass/commitlog/CommitLog-1277302220723.log)

Then, http://pastebin.com/YQA0mpRG

It's around 16:50 that cassandra writes stop timing out.  Some writes
are getting through during this 10 minutes, but they shouldn't be
enough to cause the index memtables to flush.

Thanks,

Sean






On Wed, Jun 23, 2010 at 3:30 PM, Benjamin Black <b@b3k.us> wrote:
> Are you seeing any sort of log messages from Cassandra at all?
>
> On Wed, Jun 23, 2010 at 2:26 PM, Sean Bridges <sean.bridges@gmail.com> wrote:
>> We were running a load test against a single 0.6.2 cassandra node.  24
>> hours into the test,  Cassandra appeared to be nearly frozen for 10
>> minutes.  Our write rate went to almost 0, and we had a large number
>> of write timeouts.  We weren't swapping or gc'ing at the time.
>>
>> It looks like the problems were caused by our memtables flushing after
>> 24 hours (we have MemtableFlushAfterMinutes=1440).  Some of our column
>> families are written to infrequently so that they don't hit the flush
>> thresholds in MemtableOperationsInMillions and MemtableThroughputInMB.
>>  After 24 hours we had ~3000 commit log files.
>>
>> Is this flushing causing Cassandra to become unresponsive?  I would
>> have thought Cassandra could flush in the background without blocking
>> new writes.
>>
>> Thanks,
>>
>> Sean
>>
>

Mime
View raw message