Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6CC9A200BE7 for ; Tue, 6 Dec 2016 07:06:32 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 6B7E9160B21; Tue, 6 Dec 2016 06:06:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B587C160B18 for ; Tue, 6 Dec 2016 07:06:31 +0100 (CET) Received: (qmail 13134 invoked by uid 500); 6 Dec 2016 06:06:30 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 13122 invoked by uid 99); 6 Dec 2016 06:06:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2016 06:06:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 546401AB2A5 for ; Tue, 6 Dec 2016 06:06:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.38 X-Spam-Level: ** X-Spam-Status: No, score=2.38 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=siftscience.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 0LCErMcPP-Kj for ; Tue, 6 Dec 2016 06:06:25 +0000 (UTC) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 0284B5F299 for ; Tue, 6 Dec 2016 06:06:24 +0000 (UTC) Received: by mail-wm0-f41.google.com with SMTP id f82so114225928wmf.1 for ; Mon, 05 Dec 2016 22:06:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siftscience.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=aR/PJ6Cf0B/a8/JWhbiYqvIcvSARPR4oW7u14WNofWQ=; b=S1XUo1Yi1D6MX566qeAdAedBVD2HNWDNL5q0xWakfSRF6QhZkr4gYn8H7lithlIau2 a5ramYRg8Hf/vODvb4bP61Y+Y/MRyuKB9rMiJy2jRbIeMJPoOAUJNDfil87zH9Ypae2R 1Pg3+zJ0meB7yg1dn014RtyAahdCgKXz3Asms= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=aR/PJ6Cf0B/a8/JWhbiYqvIcvSARPR4oW7u14WNofWQ=; b=FjRFeubAd7RK8sU7dMhqtmu2l4dtg6c15ONr2LHlP1k2MflHr3eH1G2SXrHcA8F8jn fk0p1OUbs9QLa4NqO4SEz3EtqilPw2MdowW7qgD2diE6DghcYz/pBiBE7/81YJKIsVMD blgqfGjdp1egyQp0gBKkloTY7y+CRel4jHqjm5RU4gxjrK+2QCkHqKxR90rEYKiPeTWr 5r4DTLqZLmF7BkyA2896eL6QasQmxGYCOc6MXGUL7XD0cFe/MojyvdYMWWLg3mghei0X 6CYLbPxGglaFejmR1ZL8LH7se4BPenjuIWgK4az/mtadsmy+G4LC8rF1LLkw1/zYU5cw jLBQ== X-Gm-Message-State: AKaTC03yO1IbLuEgUwnMUjgRQRAq9FYA/yJ22bzW5Koqs4KuF5s8OnbPkoPua0a+W3wqsOAZyG1OxZlaIvuygsum X-Received: by 10.28.17.205 with SMTP id 196mr989293wmr.78.1481004382263; Mon, 05 Dec 2016 22:06:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.25.196 with HTTP; Mon, 5 Dec 2016 22:06:21 -0800 (PST) In-Reply-To: References: From: Timothy Brown Date: Mon, 5 Dec 2016 22:06:21 -0800 Message-ID: Subject: Re: High CPU Utilization by meta region To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=001a1146df0e1dcbaa0542f7347d archived-at: Tue, 06 Dec 2016 06:06:32 -0000 --001a1146df0e1dcbaa0542f7347d Content-Type: text/plain; charset=UTF-8 We discovered that we were running the canary on all of our RegionServers. These were reading from meta and causing the CPU usage. When we stopped the canary, the CPU utilization on the RS that only contained the meta region looked like the usage on the other RegionServers. On Thu, Dec 1, 2016 at 8:43 PM, Stack wrote: > On Thu, Dec 1, 2016 at 3:48 PM, Timothy Brown wrote: > > > > > > > > > > If you turn on RPC-level TRACE logging for a minute or so, anything about > > > the client addresses that seems interesting? > > > > > > Nothing seemed interesting to me but you may have a different opinion. > > Here's the logs http://pastebin.com/FE8qVNH4 > > > > > Yeah. Nothing in there. There is no TRACE logging in there. > > St.Ack > --001a1146df0e1dcbaa0542f7347d--