incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: 10 minute cassandra pause
Date Thu, 24 Jun 2010 22:06:09 GMT
Right, I think there must be some other factor here because neither
the flush nor discard of obsolete commitlog segments block writes.

On Wed, Jun 23, 2010 at 7:05 PM, Sean Bridges <sean.bridges@gmail.com> wrote:
> I see about 3000 lines of,
>
> INFO [COMMIT-LOG-WRITER] 2010-06-23 16:40:29,107 CommitLog.java (line
> 412) Discarding obsolete commit
> log:CommitLogSegment(/data1/cass/commitlog/CommitLog-1277302220723.log)
>
> Then, http://pastebin.com/YQA0mpRG
>
> It's around 16:50 that cassandra writes stop timing out.  Some writes
> are getting through during this 10 minutes, but they shouldn't be
> enough to cause the index memtables to flush.
>
> Thanks,
>
> Sean
>
>
>
>
>
>
> On Wed, Jun 23, 2010 at 3:30 PM, Benjamin Black <b@b3k.us> wrote:
>> Are you seeing any sort of log messages from Cassandra at all?
>>
>> On Wed, Jun 23, 2010 at 2:26 PM, Sean Bridges <sean.bridges@gmail.com> wrote:
>>> We were running a load test against a single 0.6.2 cassandra node.  24
>>> hours into the test,  Cassandra appeared to be nearly frozen for 10
>>> minutes.  Our write rate went to almost 0, and we had a large number
>>> of write timeouts.  We weren't swapping or gc'ing at the time.
>>>
>>> It looks like the problems were caused by our memtables flushing after
>>> 24 hours (we have MemtableFlushAfterMinutes=1440).  Some of our column
>>> families are written to infrequently so that they don't hit the flush
>>> thresholds in MemtableOperationsInMillions and MemtableThroughputInMB.
>>>  After 24 hours we had ~3000 commit log files.
>>>
>>> Is this flushing causing Cassandra to become unresponsive?  I would
>>> have thought Cassandra could flush in the background without blocking
>>> new writes.
>>>
>>> Thanks,
>>>
>>> Sean
>>>
>>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message