Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: about FlushWriter  "All time blocked"
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: 
 <CCEC9D638518984EA83548728FAC25B239C9B562FD@EXCH07.chinacache.local>
Date: Wed, 26 Jun 2013 17:29:28 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <75BBE118-E9EC-437E-9F4C-1D3364216AC4@thelastpickle.com>
References: 
 <CCEC9D638518984EA83548728FAC25B239C9B562FD@EXCH07.chinacache.local>
To: user@cassandra.apache.org

> FlushWriter                       0         0            191         0 =
               12

This means there were 12 times the code wanted to put an memtable in the =
queue to be flushed to disk but the queue was full.=20

The length of this queue is controlled by the memtable_flush_queue_size =
https://github.com/apache/cassandra/blob/cassandra-1.2/conf/cassandra.yaml=
#L299 and memtable_flush_writers .

When this happens an internal lock around the commit log is held which =
prevents writes from being processed.=20

In general it means the IO system cannot keep up. It can sometimes =
happen when snapshot is used as all the CF's are flushed to disk at =
once. I also suspect it happens sometimes when a commit log segment is =
flushed and their are a lot of dirty CF's. But i've never proved it.=20

Increase memtable_flush_queue_size following the help in the yaml file. =
If you do not use secondary indexes are you using snapshot?

Hope that helps.=20
A
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 24/06/2013, at 3:41 PM, yue.zhang <yue.zhang@chinacache.com> wrote:

> 3 node
> cent os
> CPU 8core memory 32GB
> cassandra 1.2.5
> my scenario: many counter incr, every node has one client program, =
performance is 400 wps /every clicent (it=E2=80=99s so slowly)
> =20
> my question=EF=BC=9A
> =C3=98  nodetool tpstats
> ---------------------------------
> Pool Name                    Active   Pending      Completed   Blocked =
 All time blocked
> ReadStage                         0         0           8453         0 =
                0
> RequestResponseStage              0         0      138303982         0 =
                0
> MutationStage                     0         0      172002988         0 =
                0
> ReadRepairStage                   0         0              0         0 =
                0
> ReplicateOnWriteStage             0         0       82246354         0 =
                0
> GossipStage                       0         0        1052389         0 =
                0
> AntiEntropyStage                  0         0              0         0 =
                0
> MigrationStage                    0         0              0         0 =
                0
> MemtablePostFlusher               0         0            670         0 =
                0
> FlushWriter                       0         0            191         0 =
               12
> MiscStage                         0         0              0         0 =
                0
> commitlog_archiver                0         0              0         0 =
                0
> InternalResponseStage             0         0              0         0 =
                0
> HintedHandoff                     0         0             56         0 =
                0
> -----------------------------------
> FlushWriter =E2=80=9CAll time blocked=E2=80=9D=3D12=EF=BC=8CI restart =
the node=EF=BC=8Cbut no use=EF=BC=8Cit=E2=80=99s normally ?
> =20
> thx
> =20
> -heipark
> =20
> =20