Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8FC86200D16 for ; Tue, 10 Oct 2017 09:08:03 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8E2EA160BE0; Tue, 10 Oct 2017 07:08:03 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D37ED1609E5 for ; Tue, 10 Oct 2017 09:08:02 +0200 (CEST) Received: (qmail 71516 invoked by uid 500); 10 Oct 2017 07:08:01 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 71503 invoked by uid 99); 10 Oct 2017 07:08:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Oct 2017 07:08:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 6ADD6D7657 for ; Tue, 10 Oct 2017 06:29:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.629 X-Spam-Level: ** X-Spam-Status: No, score=2.629 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id jZ_U9Qgp8f6H for ; Tue, 10 Oct 2017 06:29:13 +0000 (UTC) Received: from mail-qt0-f196.google.com (mail-qt0-f196.google.com [209.85.216.196]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id DFB375F19C for ; Tue, 10 Oct 2017 06:29:12 +0000 (UTC) Received: by mail-qt0-f196.google.com with SMTP id 1so8087515qtn.3 for ; Mon, 09 Oct 2017 23:29:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=ZyRJYxaOg1YevCYPtDmUnfnm4bWqpXKMWxO5TEN85mU=; b=BWR6s2+s1XGV+T6qutXECGQbok6hK/MV7DU8qaoAy/wKevgC6wgU0qvrYMLXE9BFBL w006PhfnBsz4aWv5d0Tr0MgpqRWWo9qIShmZaGwvOjt5X13fi0+NgsDcJdc9cFegg4Tw CfFE43Wp6BtZhURZjfxyWfxAmFMXyibYb1Z6FicOvnMd5cumYZqPDbF3senH14vH7wAY HE9OCJR9xUzHQr+7rCLsNfr8ka/G9as7BIRp11BUMdCSjY3zjWFFQi/COgbrdT7FrD9x 4RbrZBxwqk/v/wGmxyQMOpvhOEDlVeqd4ZvHhzCz1o9iWsM+oa0yMp8J2Vfn5R1Ly97r ZvYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=ZyRJYxaOg1YevCYPtDmUnfnm4bWqpXKMWxO5TEN85mU=; b=B1Bokgk6bTzjReTxodWIdW7TlUBTPVq8znnENdXrOgbtgIHNd+f7cYuurvzoY7fWW+ Lvu+d0IdhKOFip3GORy5xacuOdAxDG2FvdsNniG60mFxGiKvo82XhI5s/KfyeCEqzvWj +MaZSXG4IIO3xmZE/At0/lA4WxF1UD8TUZ/o8Joe7LfYzclP2SdVl37MyWKk/VkFmEc5 HhawWvKbVV/Clep2nqoYcRbC/g/L9XB0lom+RBufxr4wFNC5L4TX4z0sRnPEzygamAlM SeuLQLxwA+brbTIpdZafMOzIdsk7bEuk0Q4hGkarJOXFHLvdXM4QKJvt1272sK8Or/uf PD+w== X-Gm-Message-State: AMCzsaXCLSD5MttMX6sdK5JJhkawtowRtcWeXdOwIC5NnmAdQYeIfjAd XzKKKsNG5Ygw+EcFfw2Fm9qJ2DyLmF7oDx520HsBVg== X-Google-Smtp-Source: AOwi7QCT2cZr9vMJOAn/RaGuxT3JJZXkLMaCcqqQK0EsjmtjAAu8Ou6JWawCB0f3Okg0On9nwcC1UPejfz9w4Y7iuNg= X-Received: by 10.200.39.83 with SMTP id h19mr13095514qth.35.1507616951762; Mon, 09 Oct 2017 23:29:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.159.81 with HTTP; Mon, 9 Oct 2017 23:29:11 -0700 (PDT) In-Reply-To: References: From: =?UTF-8?Q?Daniel_Jeli=C5=84ski?= Date: Tue, 10 Oct 2017 08:29:11 +0200 Message-ID: Subject: Re: OutOfMemoryError: Direct buffer memory on PUT To: user@hbase.apache.org Content-Type: multipart/alternative; boundary="001a11406032de0462055b2b6c17" archived-at: Tue, 10 Oct 2017 07:08:03 -0000 --001a11406032de0462055b2b6c17 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Vladimir, XXMaxDirectMemorySize is set to the default 0, which means unlimited as far as I can tell. Thanks, Daniel 2017-10-09 19:30 GMT+02:00 Vladimir Rodionov : > Have you try to increase direct memory size for server process? > -XXMaxDirectMemorySize=3D? > > On Mon, Oct 9, 2017 at 2:12 AM, Daniel Jeli=C5=84ski > wrote: > > > Hello, > > I'm running an application doing a lot of Puts (size anywhere between 0 > and > > 10MB, one cell at a time); occasionally I'm getting an error like the > > below: > > 2017-10-09 04:29:29,811 WARN [AsyncProcess] - #13368, > > table=3Dresearchplatform:repo_stripe, attempt=3D1/1 failed=3D1ops, last > > exception: java.io.IOException: com.google.protobuf.ServiceException: > > java.lang.OutOfMemoryError: Direct buffer memory on > > c169dzv.int.westgroup.com,60020,1506476748534, tracking started Mon Oct > 09 > > 04:29:29 EDT 2017; not retrying 1 - final failure > > > > After that the connection to RegionServer becomes unusable. Every > > subsequent attempt to execute Put on that connection results in > > CallTimeoutException. I only found the OutOfMemory by reducing the numb= er > > of tries to 1. > > > > The host running HBase appears to have at least a few GB of free memory > > available. Server logs do not mention anything about this error. Cluste= r > is > > running HBase 1.2.0-cdh5.10.2. > > > > Is this a known problem? Are there workarounds available? > > Thanks, > > Daniel > > > --001a11406032de0462055b2b6c17--