incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: 0.7.2 slow memtables flushing
Date Sun, 20 Feb 2011 03:03:15 GMT
I guess you'll need to binary search through the 0.7.1 changes to find
what made the difference.  I can't think of any obvious candidates.

On Sat, Feb 19, 2011 at 10:32 AM, Ivan Georgiev <yngwiie@bk.ru> wrote:
> On 19.2.2011 г. 16:43 ч., Jonathan Ellis wrote:
>>
>> Flush code didn't change between 0.7.0 and 0.7.2. There must be some
>> other variable here. Memory pressure maybe?
>
> Cloud you please elaborate on that one ?
> The conditions are exactly the same for the test with 0.7.0 and 0.7.2.
> By the way, 0.7.1 tests are similar to 0.7.2, while 0.7 early betas to 0.7.0
> is fine.
>
> Ivan
>
> On 19.2.2011 г. 16:43 ч., Jonathan Ellis wrote:
>>
>> Flush code didn't change between 0.7.0 and 0.7.2. There must be some
>> other variable here. Memory pressure maybe?
>>
>> On Sat, Feb 19, 2011 at 8:16 AM, Ivan Georgiev<yngwiie@bk.ru>  wrote:
>>>
>>> Hi,
>>>
>>> I am testing 0.7.2 on a Windows 2003 x64 system(one node) and I am having
>>> the following problem.
>>> My insertion speed is relatively slow, so the memtables do not get full
>>> and
>>> the actual flushing is triggered by memtable_flush_after_mins, this
>>> happens
>>> on the hour mark. My problem with 0.7.2 is that when that happens, the
>>> cpu
>>> spikes to 25% overall usage (x4 Xeon) and the operation takes anywhere
>>> from
>>> 2 to 4 minutes, leaving the node not responding during that time. This
>>> has
>>> forced to me to increase the rpc timeout option to beyond what i feel
>>> comfortable with.
>>> I have run multiple tests with 0.7.0 and 0.7.2 with the same dataset and
>>> the
>>> results are consistent. During the same operation 0.7.0 takes about 10
>>> seconds to complete vs. 2 to 4 minutes for 0.7.2. I am attaching a log
>>> with
>>> the timestamps from one such flushing of 0.7.2. Please let me know if
>>> there
>>> is anything i can do to speed up and get results similar to 0.7.0.
>>>
>>> Regards:
>>> Ivan
>>>
>>> This is the log of the operations which took most time during the flush
>>> operation. Using 0.7.0, with the same number of operations the flushing
>>> takes less than 10 seconds.
>>>
>>> INFO 01:36:44,906 Enqueuing flush of
>>> Memtable-ArchiveFiles.6f776e65724944@1225856921(1619949 bytes, 34467
>>> operations)
>>> INFO 01:37:47,375 Completed flushing
>>> C:\Storage\data\Storage\ArchiveFiles.68617368-f-3-Data.db (5549187 bytes)
>>> INFO 01:37:47,375 Writing
>>> Memtable-ArchiveFiles.6e616d65@978152661(1619949
>>> bytes, 34467 operations)
>>> INFO 01:37:47,375 Enqueuing flush of
>>> Memtable-ArchiveFiles.706172656e74466f6c6465724944@2097700961(1619949
>>> bytes,
>>> 34467 operations)
>>> INFO 01:38:51,343 Completed flushing
>>> C:\Storage\data\Storage\ArchiveFiles.6e616d65-f-3-Data.db (3820265 bytes)
>>> INFO 01:38:51,343 Writing
>>> Memtable-ArchiveFiles.6f776e65724944@1225856921(1619949 bytes, 34467
>>> operations)
>>>
>>
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message