Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 19134 invoked from network); 26 Jul 2010 21:50:01 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 Jul 2010 21:50:01 -0000 Received: (qmail 44992 invoked by uid 500); 26 Jul 2010 21:49:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 44964 invoked by uid 500); 26 Jul 2010 21:49:59 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 44956 invoked by uid 99); 26 Jul 2010 21:49:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:49:59 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dathanvp@gmail.com designates 74.125.83.172 as permitted sender) Received: from [74.125.83.172] (HELO mail-pv0-f172.google.com) (74.125.83.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:49:54 +0000 Received: by pvd12 with SMTP id 12so296655pvd.31 for ; Mon, 26 Jul 2010 14:49:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=T/MzOuvlWDG87nHbiUndnyN0dNBA1h9s+AKJkFeBoUA=; b=GZTAQ7cIo8YdwJv8BfCmtcfc+YbjO75oe3c4vZwN7I3zAx+cCVvOcUg1xhgK1H/+a3 VY5bxh2rXryQqEF4cMR/lHknDIQWduj2NWGiRuD+p2sp4QY6zck5c9Ujk2WLs/I1NyXv jO5+9zrV3V7EqzpZvLAY5l25zKB7/6zjoy9bo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=ldnQGpaC96dRcvVtDZaHMfb9sGh7Cca4uW/nV+0SH1K0agJcRU6QYWxhCmicTvfIgK xdRwjip0WiNfqsF8AeKbKfAYgHu+FnI1e+ZEZtAOJghkql6ZDoTJNhPXqc05l0u4K9m8 fFOrxaqde/dIr965t0YC35b2ULSrbEcbSmbvc= MIME-Version: 1.0 Received: by 10.142.192.4 with SMTP id p4mr9614694wff.340.1280180973408; Mon, 26 Jul 2010 14:49:33 -0700 (PDT) Received: by 10.142.73.14 with HTTP; Mon, 26 Jul 2010 14:49:33 -0700 (PDT) In-Reply-To: References: Date: Mon, 26 Jul 2010 14:49:33 -0700 Message-ID: Subject: Re: what causes MESSAGE-DESERIALIZER-POOL to spike From: Dathan Pattishall To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0cd28c902b15ad048c515c35 --000e0cd28c902b15ad048c515c35 Content-Type: text/plain; charset=ISO-8859-1 Okay so why would the pending be so large? I looked at iostat and the diskload is not bad at all. The service times /etc are all good. If it's not disk based I/O then is it thread thrashing? If so which thread pool should I monitor more closely. MESSAGE-DESERIALIZER-POOL:1 which would indicate there is one thread. If the design dictated one-thread I would assume that its a service that should not have a pending queue. Maybe there is a limitation in Cassandra Name: MESSAGE-DESERIALIZER-POOL:1 State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3c48e8f5 Total blocked: 10 Total waited: 5,427,548 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:619) shows that there is a huge amount of waiting. Is there a tunable that I can raise to remove this queue? Is it okay to have waited this high? If not what is a bad threshold? On Mon, Jul 26, 2010 at 1:31 PM, Peter Schuller wrote: > > Hpw does one set/get the read-concurrency? Also which read-concurrency, > for > > which pool? According to the threads tab in jconsole read-concurrency > could > > be among different pools. There are row-read-stag threads, lots of write > > threads, row mutation stage etc. > > Short version: Ignoremy concurrency speculation as it doesn't apply. > > Long version: > > I meant the ConcurrentReads settings in > storage-conf.xml/cassandra.yaml (depending on version), but I was > speculating. > > Having a look now, I presume it's not set to 1 because a minimum of 2 > seems to be required (DatabaseDescriptor.java). And since you ask > where I suspect it's at its default value of 8 anyway. > > Also, looking at StageManager it appears this is applied to > ROW-READ-STAGE so is indeed unrelated to MESSAGE-DESERIALIZER-POOL. > > MESSAGE-DESERIALIZER-POOL seems to be unrelated to disk I/O > alltogether and seems to be used by MessagingService.java. So - please > ignore my read concurrency speculation as it doesn't apply. > > -- > / Peter Schuller > --000e0cd28c902b15ad048c515c35 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Okay so why would the pending be so large? I looked at iostat and the diskl= oad is not bad at all. The service times /etc are all good.
If it's = not disk based I/O then is it thread thrashing? If so which thread pool sho= uld I monitor more closely.


MESSAGE-DESERIALIZER-POOL:1

which would indicate there is on= e thread. If the design dictated one-thread I would assume that its a servi= ce that should not have a pending queue. Maybe there is a limitation in Cas= sandra

Name: MESSAGE-DESERIALIZER-POOL:1
State: WAITING on java.util.concur= rent.locks.AbstractQueuedSynchronizer$ConditionObject@3c48e8f5
Total blo= cked: 10=A0 Total waited: 5,427,548

Stack trace:
sun.misc.Unsafe= .park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.u= til.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Abstr= actQueuedSynchronizer.java:1925)
java.util.concurrent.LinkedBlockingQueu= e.take(LinkedBlockingQueue.java:358)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947= )
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:907)
java.lang.Thread.run(Thread.java:619)



shows tha= t there is a huge amount of waiting. Is there a tunable that I can raise to= remove this queue? Is it okay to have waited this high? If not what is a b= ad threshold?
=A0

On Mon, Jul 26, 2010 at 1:31 PM, Pete= r Schuller <peter.schuller@infidyne.com> wrote:
> Hpw does one set/get the read-concurrency? Also whic= h read-concurrency, for
> which pool? According to the threads tab in jconsole read-concurrency = could
> be among different pools. There are row-read-stag threads, lots of wri= te
> threads, row mutation stage etc.

Short version: Ignoremy concurrency speculation as it doesn't app= ly.

Long version:

I meant the ConcurrentReads settings in
storage-conf.xml/cassandra.yaml (depending on version), but I was
speculating.

Having a look now, I presume it's not set to 1 because a minimum of 2 seems to be required (DatabaseDescriptor.java). And since you ask
where I suspect it's at its default value of 8 anyway.

Also, looking at StageManager it appears this is applied to
ROW-READ-STAGE so is indeed unrelated to MESSAGE-DESERIALIZER-POOL.

MESSAGE-DESERIALIZER-POOL seems to be unrelated to disk I/O
alltogether and seems to be used by MessagingService.java. So - please
ignore my read concurrency speculation as it doesn't apply.

--
/ Peter Schuller

--000e0cd28c902b15ad048c515c35--