Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5A2DB91EF for ; Fri, 16 Dec 2011 23:31:53 +0000 (UTC) Received: (qmail 32596 invoked by uid 500); 16 Dec 2011 23:31:51 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 32562 invoked by uid 500); 16 Dec 2011 23:31:51 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 32554 invoked by uid 99); 16 Dec 2011 23:31:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Dec 2011 23:31:51 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of homer.strong@gmail.com designates 209.85.210.41 as permitted sender) Received: from [209.85.210.41] (HELO mail-pz0-f41.google.com) (209.85.210.41) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Dec 2011 23:31:44 +0000 Received: by dakl33 with SMTP id l33so3289303dak.14 for ; Fri, 16 Dec 2011 15:31:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vT7xQwpTvD1pNcVRt/R7COkWNH1XL2TVo8tYB6dyjZY=; b=BgA7lUdXWN5thRhM+Lx3IIs3bSt4AScbYyprViafKSVaiKXmmcjds2pxOCkrRHgMoq 55GjRdnbcFUNHZUJYwuJdfr1blHZHdrB4lYyh3TGk9E+0fFF9Oih1XuoFDif7DWUorcI vw9lalciT1G4YbXkMnNVlj1GlkRFhwG0y5xiQ= MIME-Version: 1.0 Received: by 10.68.72.67 with SMTP id b3mr19190504pbv.48.1324078284163; Fri, 16 Dec 2011 15:31:24 -0800 (PST) Received: by 10.68.51.98 with HTTP; Fri, 16 Dec 2011 15:31:24 -0800 (PST) In-Reply-To: References: Date: Fri, 16 Dec 2011 15:31:24 -0800 Message-ID: Subject: Re: RS memory leak? From: Homer Strong To: user@hbase.apache.org Content-Type: text/plain; charset=ISO-8859-1 Thanks for the response! To add to our problem's description: it doesn't seem like an absolute number of regions that triggers the memory overuse, we've seen it happen now with a wide range of region counts. > Just opening regions, it does this? Yes. > No load? Very low load, no requests. > No swapping? Swapping is disabled. > Bring up more xlarge instances and see if gets you off the ground? > Then work on getting your number of regions down in number? We'll try this and get back in a couple minutes! On Fri, Dec 16, 2011 at 3:21 PM, Stack wrote: > On Fri, Dec 16, 2011 at 1:57 PM, Homer Strong wrote: >> Whenever a RS is assigned a large (> 500-600) number of regions, the >> heap usage grows without bound. Then the RS constantly GCs and must be >> killed. >> > > Just opening regions, it does this? > > No load? > > No swapping? > > What JVM and what args for JVM? > > >> This is with 2000 regions over 3 RSs, with 10 GB heap. RSs have EC2 >> xlarges. Master is on its own large. Datanodes and namenodes are >> adjacent to RSs and master, respectively. >> >> Looks like a memory leak? Any suggestions would be appreciated. >> > > Bring up more xlarge instances and see if gets you off the ground? > Then work on getting your number of regions down in number? > > St.Ack