Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D9024D6E7 for ; Sun, 19 Aug 2012 08:21:40 +0000 (UTC) Received: (qmail 78592 invoked by uid 500); 19 Aug 2012 08:21:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 78566 invoked by uid 500); 19 Aug 2012 08:21:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 78546 invoked by uid 99); 19 Aug 2012 08:21:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Aug 2012 08:21:37 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a42.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Aug 2012 08:21:33 +0000 Received: from homiemail-a42.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a42.g.dreamhost.com (Postfix) with ESMTP id F0E6C68C05D for ; Sun, 19 Aug 2012 01:21:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=4D6d8bo3k/N3w+oBFO4Zb0CLq1 I=; b=Sr2ubIA58DXoThPPZQR5WV521XiUo51mUNyJa4KbF43YhUykQA/wWXvFGD idVerb3NxjqorWI5Jaf6UqzuQeTEoeON0D8xdKwAh2RVJTgtAayVZX2jCShWRjHp p29wXYh5ryD9DATKg30/iX3CJ+CwdnRvXifBc5i4UBzX5u5wg= Received: from [172.16.1.10] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a42.g.dreamhost.com (Postfix) with ESMTPSA id 4296068C05B for ; Sun, 19 Aug 2012 01:21:11 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_C20EB1A0-DC8F-4113-B486-2AD2C5CD07F0" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Why the StageManager thread pools have 60 seconds keepalive time? Date: Sun, 19 Aug 2012 20:21:09 +1200 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_C20EB1A0-DC8F-4113-B486-2AD2C5CD07F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Your seeing dropped mutations reported from nodetool tpstats ?=20 Take a look at the logs. Look for messages from the MessagingService = with the pattern "{} {} messages dropped in last {}ms" They will be = followed by info about the TP stats. First would be the workload. Are you sending very big batch_mutate or = multiget requests? Each row in the requests turns into a command in the = appropriate thread pool. This can result in other requests waiting a = long time for their commands to get processed.=20 Next would be looking for GC and checking the memtable_flush_queue_size = is set high enough (check yaml for docs).=20 After that I would look at winding concurrent_writers (and I assume = concurrent_readers) back. Anytime I see weirdness I look for config = changes and see what happens when they are returned to the default or = near default. Do you have 16 _physical_ cores? Hope that helps.=20 =20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 18/08/2012, at 10:01 AM, Guillermo Winkler = wrote: > Aaron, thanks for your answer. >=20 > I'm actually tracking a problem where mutations get dropped and = cfstats show no activity whatsoever, I have 100 threads for the mutation = pool, no running or pending tasks, but some mutations get dropped none = the less. >=20 > I'm thinking about some scheduling problems but not really sure yet. >=20 > Have you ever seen a case of dropped mutations with the system under = light load? >=20 > Thanks, > Guille >=20 >=20 > On Thu, Aug 16, 2012 at 8:22 PM, aaron morton = wrote: > That's some pretty old code. I would guess it was done that way to = conserve resources. And _i think_ thread creation is pretty light = weight. >=20 > Jonathan / Brandon / others - opinions ?=20 >=20 > Cheers >=20 >=20 > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com >=20 > On 17/08/2012, at 8:09 AM, Guillermo Winkler = wrote: >=20 >> Hi, I have a cassandra cluster where I'm seeing a lot of thread = trashing from the mutation pool. >>=20 >> MutationStage:72031 >>=20 >> Where threads get created and disposed in 100's batches every few = minutes, since it's a 16 core server concurrent_writes is set in 100 in = the cassandra.yaml.=20 >>=20 >> concurrent_writes: 100 >>=20 >> I've seen in the StageManager class this pools get created with 60 = seconds keepalive time. >>=20 >> DebuggableThreadPoolExecutor -> allowCoreThreadTimeOut(true); >>=20 >> StageManager-> public static final long KEEPALIVE =3D 60; // seconds = to keep "extra" threads alive for when idle >>=20 >> Is it a reason for it to be this way?=20 >>=20 >> Why not have a fixed size pool with Integer.MAX_VALUE as keepalive = since corePoolSize and maxPoolSize are set at the same size? >>=20 >> Thanks, >> Guille >>=20 >=20 >=20 --Apple-Mail=_C20EB1A0-DC8F-4113-B486-2AD2C5CD07F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Your = seeing dropped mutations reported from nodetool tpstats = ? 

Take a look at the logs. Look for messages = from the MessagingService with the pattern "{} {} messages dropped in = last {}ms" They will be followed by info about the TP = stats.

First would be the workload. Are you = sending very big batch_mutate or multiget requests? Each row in the = requests turns into a command in the appropriate thread pool. This can = result in other requests waiting a long time for their commands to get = processed. 

Next would be looking for GC = and checking the memtable_flush_queue_size is set high enough (check = yaml for docs). 

After that I would look = at winding  concurrent_writers (and I assume concurrent_readers) = back. Anytime I see weirdness I look for config changes and see what = happens when they are returned to the default or near default.  Do = you have 16 _physical_ cores?

Hope that = helps. 
  
=
http://www.thelastpickle.com

On 18/08/2012, at 10:01 AM, Guillermo Winkler <gwinkler@inconcertcc.com> = wrote:

Aaron, thanks for your answer.

I'm = actually tracking a problem where mutations get dropped and cfstats show = no activity whatsoever, I have 100 threads for the mutation pool, no = running or pending tasks, but some mutations get dropped none the = less.

I'm thinking about some scheduling problems but not = really sure yet.

Have you ever seen a case of = dropped mutations with the system under light = load?

Thanks,
Guille


On = Thu, Aug 16, 2012 at 8:22 PM, aaron morton <aaron@thelastpickle.com> wrote:
That's some pretty old code. I would = guess it was done that way to conserve resources. And _i think_ thread = creation is pretty light weight.

Jonathan / Brandon / others - opinions = ? 

Cheers

<= /div>

-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 17/08/2012, at 8:09 AM, Guillermo Winkler <gwinkler@inconcertcc.com> = wrote:

Hi, I have a cassandra cluster = where I'm seeing a lot of thread trashing from the mutation pool.

MutationStage:72031

Where threads get created and disposed in = 100's batches every few minutes, since it's a 16 core server = concurrent_writes is set in 100 in the cassandra.yaml.

concurrent_writes: 100

I've seen in the StageManager class = this pools get created with 60 seconds keepalive = time.

DebuggableThreadPoolExecutor = -> allowCoreThreadTimeOut(true);

StageManager-> pub= lic static final long KEEPALIVE =3D 60; // seconds to keep "extra" = threads alive for when idle

Is it a reason for it to be this way? 

Why = not have a fixed size pool with Integer.MAX_VALUE as keepalive since = corePoolSize and maxPoolSize are set at the same = size?

Thanks,
Guille

=



= --Apple-Mail=_C20EB1A0-DC8F-4113-B486-2AD2C5CD07F0--