incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: is it possible for light-traffic CF to hold down many commit logs?
Date Fri, 23 Sep 2011 05:47:41 GMT
thanks for the input.

if that's the case, I think the solution would be to sort the CFs to
flush by a more complex criteria than just size. for example the
number of dirty commit logs that contain this CF should be considered
as a score.

Yang

On Thu, Sep 22, 2011 at 10:40 PM, Philippe <watcherfr@gmail.com> wrote:
> It sure looks like what I'm seeing on my cluster where a 100G commit lot
> partition fills up in 12 hours (0.8.x)
>
> Le 23 sept. 2011 03:45, "Yang" <teddyyyy123@gmail.com> a écrit :
>> in 1.0.0 we don't have memtable_throughput for each individual CF ,
>> and instead
>> which memtable/CF to flush is determined by "largest
>> getTotalMemtableLiveSize() ".
>> (MeteredFlusher.java line 81)
>>
>>
>> what would happen in the following case ? : I have only 2 CF, the
>> traffic for one CF is 1000 times that
>> of the second CF,
>> so the high-traffic CF constantly triggers total mem threshold , and
>> every time, the busy CF is flushed.
>>
>> but the light-traffic CF is never flushed ( well, until we have
>> flushed about 1000 times the busy CF),
>> now we are left with many commit logs , each of them containing a few
>> entries for the light-traffic table. we have to keep these commit logs
>> because these entries are not flushed to sstable yet.
>>
>> then there are 2 problems: 1) to persist the few records from the
>> light-traffic CF, you have to keep 1000 times the commit logs
>> necessary, taking up disk space 2) when you do a recover on server
>> restart, you'll have to read through all those commit logs .
>>
>> does the above hypothesis sound right?
>>
>> Thanks
>> Yang
>

Mime
View raw message