Return-Path: 
 <user-return-19666-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CAA217393
	for <apmail-hadoop-user-archive@minotaur.apache.org>;
 Wed,  6 May 2015 14:05:38 +0000 (UTC)
Received: (qmail 85772 invoked by uid 500); 6 May 2015 14:05:32 -0000
Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org
Received: (qmail 85668 invoked by uid 500); 6 May 2015 14:05:32 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 85653 invoked by uid 99); 6 May 2015 14:05:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2015 14:05:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: message received from 54.164.171.186
 which is an MX secondary for user@hadoop.apache.org)
Received: from [54.164.171.186] (HELO mx1-us-east.apache.org) (54.164.171.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2015 14:05:25 +0000
Received: from mail-la0-f45.google.com (mail-la0-f45.google.com
 [209.85.215.45])
	by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with
 ESMTPS id E594743E67
	for <user@hadoop.apache.org>; Wed,  6 May 2015 14:05:04 +0000 (UTC)
Received: by lagv1 with SMTP id v1so8129312lag.3
        for <user@hadoop.apache.org>; Wed, 06 May 2015 07:02:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=9vJvz/eWgX61xnVfT8QcpZQD4AF4sdTnhyAbOw7cnhg=;
        b=JQmtD9C+xM8wibuXQDZ1edtL2X23gVpWv88kD85+PFJUMP3n3VddURtegR2PIoEiio
         xfWGqQDASNf8TdIJVJFK0rk+JOHVP3g6fa9ppV8sFXAHsn4iPTQmQiQkwYmZp6ZsBEQB
         6oHIp1K3VopjXNpdLx8jkJ1LEhL74hzEyfyRYl5XZUSSBIjCPI9yVxJ6Qwi5nXkREbAS
         jszz4JRpyz/GQdVRxYrG38fa369FJ9ip+TKxt0Oe8dHE9XWcEqbA7aifUFhE081fJy/D
         Hx84yZEgteSuRaOcD1SCADmRsoIb8fcFEnZONg26PnqjUou9PoJlYiDU2BWpDmhfo6ar
         +hxA==
MIME-Version: 1.0
X-Received: by 10.152.87.204 with SMTP id ba12mr29138326lab.35.1430920968698;
 Wed, 06 May 2015 07:02:48 -0700 (PDT)
Received: by 10.114.59.104 with HTTP; Wed, 6 May 2015 07:02:48 -0700 (PDT)
Date: Wed, 6 May 2015 19:02:48 +0500
Message-ID: 
 <CAABizCQN0ivQ0ogMOyEB89Z+droGvxk2tjcnZ-o8bBWEg=D=ew@mail.gmail.com>
Subject: BlockPoolSliceScanner and GC
From: =?UTF-8?B?0KHQtdGA0LPQtdC5INCa0LDQt9Cw0LrQvtCy?= <ss.fl1x@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11c364a60a815605156a40be
X-Virus-Checked: Checked by ClamAV on apache.org

--001a11c364a60a815605156a40be
Content-Type: text/plain; charset=UTF-8

Hi everyone!


We've been investigating some unpredictably long GC pauses in Hadoop
Datanode (2.5.1, Oracle JRE8_25) and noticed strange behaviour in
DataBlockScanner.

DataBlockScanner initiates block verification tasks during DN startup and
every 504 hours after. It holds some structures in memory heap (HashMap,
TreeSet & LightWeightGSet) which, in our case, are around 330Mb total (over
1.5m blocks). Are these structures useful after block verification task
completes? Tenuring and promoting these objects to OldGen causes some
terrible STW pauses we would like to avoid.


Kind regards,
Sergey Kazakov

--001a11c364a60a815605156a40be
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi everyone!<div><br></div><div><br></div><div>We&#39;ve b=
een investigating some unpredictably long GC pauses in Hadoop Datanode (2.5=
.1, Oracle JRE8_25) and noticed strange behaviour in DataBlockScanner.</div=
><div><br></div><div>DataBlockScanner initiates block verification tasks du=
ring DN startup and every 504 hours after. It holds some structures in memo=
ry heap (HashMap, TreeSet &amp; LightWeightGSet) which, in our case, are ar=
ound 330Mb total (over 1.5m blocks). Are these structures useful after bloc=
k verification task completes? Tenuring and promoting these objects to OldG=
en causes some terrible STW pauses we would like to avoid.</div><div><br cl=
ear=3D"all"><div><br></div>Kind regards,
</div><div>Sergey Kazakov</div></div>

--001a11c364a60a815605156a40be--