Return-Path: 
 <user-return-43127-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 087FF1154D
	for <apmail-hbase-user-archive@www.apache.org>;
 Tue, 13 May 2014 23:56:21 +0000 (UTC)
Received: (qmail 41149 invoked by uid 500); 13 May 2014 19:56:20 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 41065 invoked by uid 500); 13 May 2014 19:56:20 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 41050 invoked by uid 99); 13 May 2014 19:56:20 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 May 2014 19:56:20 +0000
X-ASF-Spam-Status: No, hits=2.8 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.216.174 as permitted sender)
Received: from [209.85.216.174] (HELO mail-qc0-f174.google.com)
 (209.85.216.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 May 2014 19:56:16 +0000
Received: by mail-qc0-f174.google.com with SMTP id x13so1148362qcv.19
        for <user@hbase.apache.org>; Tue, 13 May 2014 12:55:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:sender:in-reply-to:references:date:message-id:subject
         :from:to:content-type;
        bh=5Z12o6+cDizUc08b8AOYcarwIW8XALcFbA0ZGK+ln14=;
        b=PSZWv/fKvFvU7sH66FlZqExI+WM79YdEruH2t7RqVnUu/T0aqjvn6GmzytXLjRH/nL
         fSiNfVlfF5d/EU/w+LKde5jvP1NkOSwMfhopUb7C9hHOjjEpd4IzQDrR9AHLxJrD3uSq
         58RQEc4SxW9yd4UIXYaBB+L1HoWkqVp+mq4OlQSNUFYJU5F2gVPGdh1Tx+yMYxKj58ib
         9XVDdoLw+Qr4NNoQnp7TSjO+bCYFcZbA8mItGjz57MGhAAEDAc4w8CcCwwU5kQL9ME1S
         aqoGNPBP7bPCqPG3dx5P8OA8qt7+rQjx2Z+APned32hsOlAKREHnCBpSU0WfRru2eRIk
         8/Jg==
MIME-Version: 1.0
X-Received: by 10.140.87.243 with SMTP id r106mr48139648qgd.75.1400010953382;
 Tue, 13 May 2014 12:55:53 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.96.183.230 with HTTP; Tue, 13 May 2014 12:55:53 -0700 (PDT)
In-Reply-To: 
 <CAKrkF=tMUCje-ED2ia3KqNorsKvCSTKP7R_JGv4kr9YHDZ37fQ@mail.gmail.com>
References: 
 <CAKrkF=uuPVqMoa-Ew9q8BssCgX_reZ6WB=jwg2aSNjxVyKVjCQ@mail.gmail.com>
	<CAKrkF=skrhvqUv2TyDZ65k4h+E09_2d1SKcrK06bVkXUyj6ryg@mail.gmail.com>
	<DC5EBE7F3610EB4CA5C7E92D76873E15187CA57C58@exchange2007.carrieriq.com>
	<CAKrkF=tMUCje-ED2ia3KqNorsKvCSTKP7R_JGv4kr9YHDZ37fQ@mail.gmail.com>
Date: Tue, 13 May 2014 12:55:53 -0700
X-Google-Sender-Auth: cyOt-4O1Dtmwbdj47shpOFSlECw
Message-ID: 
 <CADcMMgFS0QUC+-aZ4NFRtC9pTtnpHTCwU2SHX-D=JnLnW9uROQ@mail.gmail.com>
Subject: Re: RPC Client OutOfMemoryError Java Heap Space
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c11e748f12d304f94d7321
X-Virus-Checked: Checked by ClamAV on apache.org

--001a11c11e748f12d304f94d7321
Content-Type: text/plain; charset=UTF-8

A patch for the refguide would be great, perhaps in the troubleshooting
mapreduce section here http://hbase.apache.org/book.html#trouble.mapreduce?
St.Ack


On Tue, May 13, 2014 at 7:07 AM, Geovanie Marquez <
geovanie.marquez@gmail.com> wrote:

> The following property does exactly what I wanted our environment to do. I
> had a 4GiB Heap and ran the job and no jobs failed. Then I dropped our
> cluster heap to 1GiB and reran the same resource intensive task.
>
> This property must be added to the "HBase Service Advanced Configuration
> Snippet (Safety Valve) for hbase-site.xml"
>
> <property>
> <name>hbase.client.scanner.max.result.size</name>
> <value>67108864</value>
> </property>
>
> We noted that 64MiB would be enough, but we also experimented 128MiB. I may
> do a write-up and elaborate some more on this.
>
>
> On Mon, May 12, 2014 at 1:38 PM, Vladimir Rodionov
> <vrodionov@carrieriq.com>wrote:
>
> > All your OOME are on the client side (map task). Your map tasks need more
> > heap.
> > Reduce # of map tasks and increase max heap size per map task.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: Geovanie Marquez [geovanie.marquez@gmail.com]
> > Sent: Thursday, May 08, 2014 2:35 PM
> > To: user@hbase.apache.org
> > Subject: Re: RPC Client OutOfMemoryError Java Heap Space
> >
> > sorry didn't include version
> >
> > CDH5 version - CDH-5.0.0-1.cdh5.0.0.p0.47
> >
> >
> > On Thu, May 8, 2014 at 5:32 PM, Geovanie Marquez <
> > geovanie.marquez@gmail.com
> > > wrote:
> >
> > > Hey group,
> > >
> > > There is one job that scans HBase contents and is really resource
> > > intensive using all resources available to yarn (under Resource
> Manager).
> > > In my case, that is 8GB. My expectation here is that a properly
> > configured
> > > cluster would kill the application or degrade the application
> performance
> > > but never ever take a region server down. This is intended to be a
> > > multi-tenant environment where developers may submit jobs at will and I
> > > would want a configuration where the cluster services are not exited in
> > > this way because of memory.
> > >
> > > The simple solution here, is to change the way the job consumes
> resources
> > > so that when run it is not so resource greedy. I want to understand
> how I
> > > can mitigate this situation in general.
> > >
> > > **It FAILS with the following config:**
> > > The RPC client has 30 handlers
> > > write buffer of 2MiB
> > > The RegionServer heap is 4GiB
> > > Max Size of all memstores is 0.40 of total heap
> > > HFile Block Cache Size is 0.40
> > > Low watermark for memstore flush is 0.38
> > > HBase Memstore size is 128MiB
> > >
> > > **Job still FAILS with the following config:**
> > > Everything else the same except
> > > The RPC client has 10 handlers
> > >
> > > **Job still FAILS with the following config:**
> > > Everything else the same except
> > > HFile Block Cache Size is 0.10
> > >
> > >
> > > When this runs I get the following error stacktrace:
> > > #
> > > #How do I avoid this via configuration.
> > > #
> > >
> > > java.lang.OutOfMemoryError: Java heap space
> > >       at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1100)
> > >       at
> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:721)
> > > 2014-05-08 16:23:54,705 WARN [IPC Client (1242056950) connection to
> > c1d001.in.wellcentive.com/10.2.4.21:60020 from hbase]
> > org.apache.hadoop.ipc.RpcClient: IPC Client (1242056950) connection to
> > c1d001.in.wellcentive.com/10.2.4.21:60020 from hbase: unexpected
> > exception receiving call responses
> > > #
> > >
> > > ###Yes, there was an RPC timeout this is what is killing the server
> > because the timeout is eventually (1minute later) reached.
> > >
> > > #
> > >
> > > java.lang.OutOfMemoryError: Java heap space
> > >       at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1100)
> > >       at
> > org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:721)
> > > 2014-05-08 16:23:55,319 INFO [main]
> > org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: recovered from
> > org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> > OutOfOrderScannerNextException: was there a rpc timeout?
> > >       at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:384)
> > >       at
> >
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:194)
> > >       at
> >
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
> > >       at
> >
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
> > >       at
> >
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> > >       at
> >
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> > >       at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > >       at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> > >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > >       at java.security.AccessController.doPrivileged(Native Method)
> > >       at javax.security.auth.Subject.doAs(Subject.java:415)
> > >       at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> > >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> > >
> > > #
> > >
> > > ## Probably caused by the OOME above
> > >
> > > #
> > >
> > > Caused by:
> > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> Expected
> > nextCallSeq: 1 But the nextCallSeq got from client: 0;
> request=scanner_id:
> > 5612205039322936440 number_of_rows: 10000 close_scanner: false
> > next_call_seq: 0
> > >       at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3018)
> > >       at
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
> > >       at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
> > >       at
> > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
> > >
> > >
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
> >
>

--001a11c11e748f12d304f94d7321--