Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D026C920F for ; Thu, 24 Nov 2011 17:42:51 +0000 (UTC) Received: (qmail 27220 invoked by uid 500); 24 Nov 2011 17:42:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 27186 invoked by uid 500); 24 Nov 2011 17:42:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 27178 invoked by uid 99); 24 Nov 2011 17:42:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Nov 2011 17:42:49 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of scode@scode.org designates 209.85.215.172 as permitted sender) Received: from [209.85.215.172] (HELO mail-ey0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Nov 2011 17:42:42 +0000 Received: by eabm6 with SMTP id m6so380441eab.31 for ; Thu, 24 Nov 2011 09:42:21 -0800 (PST) MIME-Version: 1.0 Received: by 10.180.81.163 with SMTP id b3mr29317438wiy.20.1322156540924; Thu, 24 Nov 2011 09:42:20 -0800 (PST) Sender: scode@scode.org Received: by 10.180.103.198 with HTTP; Thu, 24 Nov 2011 09:42:20 -0800 (PST) X-Originating-IP: [71.202.44.53] In-Reply-To: <4ECA2E22.9020801@ic-drei.de> References: <4ECA2E22.9020801@ic-drei.de> Date: Thu, 24 Nov 2011 09:42:20 -0800 X-Google-Sender-Auth: 7e0PkZ6bs_9Xs5XipA5gdllv1gU Message-ID: Subject: Re: Pending ReadStage is exploding on only one node From: Peter Schuller To: user@cassandra.apache.org Content-Type: text/plain; charset=UTF-8 > I'm measuring a high load value on a few nodes during the update process > (which is normal), but one node keeps the high load after the process for a > long time. I would say that either the reading that you to is overloading that one node and other traffic is getting piled up as a result, or you're stomping on page cache by reading a lot from that one node (e.g. using CL.ONE) and you're then seeing readstage backed up until the page cache or row cache is warm again. In general, unless you're running at close to full CPU capacity it sounds like you're completely disk bound, and that'll show up as a huge amount of pending ReadStage. "iostat -x -k 1" should confirm it. -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)