cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: MeteredFlusher in system.log entries
Date Wed, 06 Jun 2012 18:53:04 GMT
You question was
>>>> Could you please throw light on the what conditions does
>>>> MeteredFlusher use to trigger memtable flushes.

The answer is estimates of the ratio between the live size and the serialised size of memtables
are kept. The MeteredFlusher periodically checks the serialised size of all memtables and
uses the ratio to determine if memtable_total_space_in_mb has been reached. If there is a
variation between nodes it may be that some are getting more traffic than others. 

>> So it implies that for flushing, Cassandra copies the memtables content.

> 
No

>> So does this imply that writes to column families are not stopped even
when it is being flushed?
Yes. 
In a worst case scenario writes will block if the memtable flushing cannot keep up. 

> Also, Could someone please explain how the factor of 7 comes in the
picture in this sentence

In the example (see previous para) 7 is the number of memtables the CF could have in memory
at once (forgetting about the other cf's).  

Cheers
 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 7/06/2012, at 1:08 AM, rohit bhatia wrote:

> Also, Could someone please explain how the factor of 7 comes in the
> picture in this sentence
> 
> "For example if memtable_total_space_in_mb is 100MB, and
> memtable_flush_writers is the default 1 (with one data directory), and
> memtable_flush_queue_size is the default 4, and a Column Family has no
> secondary indexes. The CF will not be allowed to get above one seventh
> of 100MB or 14MB, as if the CF filled the flush pipeline with 7
> memtables of this size it would take 98MB. "
> 
> On Wed, Jun 6, 2012 at 6:22 PM, rohit bhatia <rohit2412@gmail.com> wrote:
>> Hi..
>> 
>> the link http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/
>> mentions that "From version 0.7 onwards the worse case scenario is up
>> to CF Count + Secondary Index Count + memtable_flush_queue_size
>> (defaults to 4) + memtable_flush_writers (defaults to 1 per data
>> directory) memtables in memory the JVM at once.".
>> 
>> So it implies that for flushing, Cassandra copies the memtables content.
>> So does this imply that writes to column families are not stopped even
>> when it is being flushed?
>> 
>> Thanks
>> Rohit
>> 
>> On Wed, Jun 6, 2012 at 9:42 AM, rohit bhatia <rohit2412@gmail.com> wrote:
>>> Hi Aaron
>>> 
>>> Thanks for the link, I have gone through it. But this doesn't justify
>>> nodes of exactly same config/specs differing in their flushing
>>> frequency.
>>> The traffic on all node is same as we are using RandomPartitioner
>>> 
>>> Thanks
>>> Rohit
>>> 
>>> On Wed, Jun 6, 2012 at 12:24 AM, aaron morton <aaron@thelastpickle.com>
wrote:
>>>> See the section on memtable_total_space_in_mb here
>>>>  http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/
>>>> 
>>>> Cheers
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>> 
>>>> On 6/06/2012, at 2:27 AM, rohit bhatia wrote:
>>>> 
>>>> I am trying to understand the variance in flushes frequency in a 8
>>>> node Cassandra cluster.
>>>> All the flushes are of the same type and initiated by MeteredFlusher.java
=>
>>>> 
>>>> "INFO [OptionalTasks:1] 2012-06-05 06:32:05,873 MeteredFlusher.java
>>>> (line 62) flushing high-traffic column family CFS(Keyspace='Stats',
>>>> ColumnFamily='Minutewise_Channel_Stats') (estimated 501695882 bytes)"
>>>> [taken from system.log]
>>>> 
>>>> Number of flushes for 1 column family vary from 6 flushes per day to
>>>> 24 flushes per day among nodes of same configuration and same
>>>> hardware.
>>>> Could you please throw light on the what conditions does
>>>> MeteredFlusher use to trigger memtable flushes.
>>>> Also how accurate is the estimated size in the above logfile entry.
>>>> 
>>>> Regards
>>>> Rohit Bhatia
>>>> Software Engineer, Media.net
>>>> 
>>>> 


Mime
View raw message