incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: about FlushWriter "All time blocked"
Date Wed, 26 Jun 2013 05:29:28 GMT
> FlushWriter                       0         0            191         0               
12

This means there were 12 times the code wanted to put an memtable in the queue to be flushed
to disk but the queue was full. 

The length of this queue is controlled by the memtable_flush_queue_size https://github.com/apache/cassandra/blob/cassandra-1.2/conf/cassandra.yaml#L299
and memtable_flush_writers .

When this happens an internal lock around the commit log is held which prevents writes from
being processed. 

In general it means the IO system cannot keep up. It can sometimes happen when snapshot is
used as all the CF's are flushed to disk at once. I also suspect it happens sometimes when
a commit log segment is flushed and their are a lot of dirty CF's. But i've never proved it.


Increase memtable_flush_queue_size following the help in the yaml file. If you do not use
secondary indexes are you using snapshot?

Hope that helps. 
A
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 24/06/2013, at 3:41 PM, yue.zhang <yue.zhang@chinacache.com> wrote:

> 3 node
> cent os
> CPU 8core memory 32GB
> cassandra 1.2.5
> my scenario: many counter incr, every node has one client program, performance is 400
wps /every clicent (it’s so slowly)
>  
> my question:
> Ø  nodetool tpstats
> ---------------------------------
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         0         0           8453         0              
  0
> RequestResponseStage              0         0      138303982         0              
  0
> MutationStage                     0         0      172002988         0              
  0
> ReadRepairStage                   0         0              0         0              
  0
> ReplicateOnWriteStage             0         0       82246354         0              
  0
> GossipStage                       0         0        1052389         0              
  0
> AntiEntropyStage                  0         0              0         0              
  0
> MigrationStage                    0         0              0         0              
  0
> MemtablePostFlusher               0         0            670         0              
  0
> FlushWriter                       0         0            191         0              
 12
> MiscStage                         0         0              0         0              
  0
> commitlog_archiver                0         0              0         0              
  0
> InternalResponseStage             0         0              0         0              
  0
> HintedHandoff                     0         0             56         0              
  0
> -----------------------------------
> FlushWriter “All time blocked”=12,I restart the node,but no use,it’s normally
?
>  
> thx
>  
> -heipark
>  
>  


Mime
View raw message