cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: cassandra freezes
Date Fri, 26 Feb 2010 13:53:42 GMT
It will allow several memtables to queue up, and then it will block
further writes so it doesn't exhaust memory while it flushes.

Perhaps that is what you are seeing.  You should check the flush queue
sizes with JMX or nodeprobe tpstats.

On Fri, Feb 26, 2010 at 5:25 AM, Boris Shulman <shulmanb@gmail.com> wrote:
> What will be the implications of the fact that cassandra can't keep up
> with the write? Will the memtables be queued in memory until they are
> flushed?
>
> On Thu, Feb 25, 2010 at 4:56 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>> Are you swapping?
>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html
>>
>> otherwise there's something wrong w/ your vm (?), disk i/o doesn't
>> block incoming writes in cassandra
>>
>> On Thu, Feb 25, 2010 at 8:49 AM, Boris Shulman <shulmanb@gmail.com> wrote:
>>> I don't think it is gc related issue. There is no correlation between
>>> gc times and the freeze times. More over I don't see any gc activity
>>> that lasts for omre than o.03 sec. But there is a correlation between
>>> disk flushing operations. I've noticed that the system freezes each
>>> time when my commit log reaches 1.1G. I have 1024M memtable size so I
>>> assume this is when the data flushing occurs.
>>>
>>>
>>>
>>> On Thu, Feb 25, 2010 at 4:13 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>>> Then you should check GC timing with -Xverbose:gc option (see:
>>>> http://wiki.apache.org/cassandra/RunningCassandra for how to modify
>>>> jvm options) for a correlation.
>>>>
>>>> On Thu, Feb 25, 2010 at 8:09 AM, Boris Shulman <shulmanb@gmail.com>
wrote:
>>>>> In these tests I perform only write operations, no reads.
>>>>>
>>>>> On Thu, Feb 25, 2010 at 4:07 PM, Jonathan Ellis <jbellis@gmail.com>
wrote:
>>>>>> The only kind of "freeze" that makes sense there is your reads are
i/o
>>>>>> bound and the extra disk activity is killing you.  In that case
the
>>>>>> fix is to add more RAM, or give less to the JVM so the OS can use
more
>>>>>> for buffer cache.
>>>>>>
>>>>>> On Thu, Feb 25, 2010 at 8:01 AM, Boris Shulman <shulmanb@gmail.com>
wrote:
>>>>>>> In my case the cassandra node freezes while memtable flush operation
>>>>>>> is performed or compactation operation is performed. How can
I
>>>>>>> optimize the cassandra configuration in order to avoid this behavior?
>>>>>>> I've tried both using large memtable size (1G) and small (128M)
but in
>>>>>>> every case I have some sort of freezes when the data is flushed
to the
>>>>>>> disk.
>>>>>>>
>>>>>>> Please advice.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message