Return-Path: 
 <user-return-43583-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 872AC1140A
	for <apmail-hbase-user-archive@www.apache.org>;
 Tue, 10 Jun 2014 21:43:32 +0000 (UTC)
Received: (qmail 78440 invoked by uid 500); 10 Jun 2014 21:43:30 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 78365 invoked by uid 500); 10 Jun 2014 21:43:30 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 78354 invoked by uid 99); 10 Jun 2014 21:43:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jun 2014 21:43:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of otis.gospodnetic@gmail.com
 designates 209.85.216.179 as permitted sender)
Received: from [209.85.216.179] (HELO mail-qc0-f179.google.com)
 (209.85.216.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jun 2014 21:43:25 +0000
Received: by mail-qc0-f179.google.com with SMTP id r5so3161295qcx.38
        for <user@hbase.apache.org>; Tue, 10 Jun 2014 14:43:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=references:in-reply-to:mime-version:content-type:message-id
         :content-transfer-encoding:from:subject:date:to;
        bh=rt7lCv9H1HBbFMG8CM3DZddroxGzeSB3MmoonWxeXes=;
        b=LLevR+gPRp8YmNlrdbnSr2h3m4OYiU26MtT2hjs9cT1mErg1YVpgEJZkRW4C+3L/wR
         x2V4hUeoLH4YMD9zOEB1O6nxX0N8bbnmgIevCAQJn3A46jXkM6ibYpJkM4coQhiDdAoI
         AkfxXN5RoiFwEThSWFF3PLCkUog/MTco++OtRvW8KhXVlw3+u0Ky0Aq1LZwVZ467sFOU
         HmljsGl9UcCm2UvhJUneOpf+2Vf1FEMENz8IpWJjyU8dt/kz81M3CXuQpZcRb4TLipZz
         8EEf/xWRSjE1XinKL3DzGwcxUHBst+pwuGARE3Z3vkeU2DeA1fcIKlIDTsho/fZnjhcn
         vQ8w==
X-Received: by 10.224.127.131 with SMTP id g3mr46038897qas.31.1402436584638;
        Tue, 10 Jun 2014 14:43:04 -0700 (PDT)
Received: from ?IPv6:2600:1001:b10a:5894:b89e:8f3:d8c6:30df?
 ([2600:1001:b10a:5894:b89e:8f3:d8c6:30df])
        by mx.google.com with ESMTPSA id
 g4sm23993878qas.48.2014.06.10.14.43.03
        for <multiple recipients>
        (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Tue, 10 Jun 2014 14:43:04 -0700 (PDT)
References: 
 <CAHtRq2eeGdRx4+YZZ_csnBqOnuXD5v4y8-G_8oTJY+bmrHFbBw@mail.gmail.com>
In-Reply-To: 
 <CAHtRq2eeGdRx4+YZZ_csnBqOnuXD5v4y8-G_8oTJY+bmrHFbBw@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <00B12C12-4ABE-49BA-BA82-B248D5A5A13E@gmail.com>
Content-Transfer-Encoding: quoted-printable
X-Mailer: iPhone Mail (11D201)
From: Otis Gospodnetic <otis.gospodnetic@gmail.com>
Subject: Re: Is this a long GC pause, or something else?
Date: Tue, 10 Jun 2014 17:43:04 -0400
To: "user@hbase.apache.org" <user@hbase.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

Does it repeat?
We are seeing this with u60 oracle JVM too!  SPM shows the whole JVM blockin=
g for about 16 minutes every M minutes.=20

Otis

=20

> On Jun 10, 2014, at 2:05 PM, Tom Brown <tombrown52@gmail.com> wrote:
>=20
> Last night a regionserver in my cluster stopped responding in a timely
> manner for about 20 minutes. I know that stop-the-world GC can cause this
> type of behavior, but 20 minutes seems excessive.
>=20
> The server is a 2 core VM with 16GB of RAM, (hbase max heap is 12GB). We
> are using the latest java 7 from oracle. HDFS is provided by an Isilon
> cluster.
>=20
> The server workload is read/write: the writing process reads all rows it i=
s
> about to write, updates them if they exist, and then writes all the rows
> (replacing ones that were updated).
>=20
> The last messages before the pause were regarding an HLog roll:
>=20
> DEBUG org.apache.hadoop.hbase.regionserver.LogRoller: HLog roll requested
> INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
> getDefaultReplication
> INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
> getDefaultBlockSize
>=20
> During the next 20 minutes there were a handful of sporadic LruBlockCache
> stats messages but nothing else. After 20 minutes, normal operation resume=
d.
>=20
> Is 20 minutes for a GC pause expected given the operational load and
> machine specs? Could a GC pause include periodic log messages? If it wasn'=
t
> a GC pause, what else could it be?
>=20
> --Tom