Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 6 Aug 2013 16:02:48 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12624661.1355933765919.9675.1375804968986@arcas>
In-Reply-To: <JIRA.12624661.1355933765919@arcas>
References: <JIRA.12624661.1355933765919@arcas>
Subject: [jira] [Assigned] (HBASE-7391) Review/improve HLog compression's
 memory consumption
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HBASE-7391?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan reassigned HBASE-7391:
---------------------------------------------

    Assignee: ramkrishna.s.vasudevan
   =20
> Review/improve HLog compression's memory consumption
> ----------------------------------------------------
>
>                 Key: HBASE-7391
>                 URL: https://issues.apache.org/jira/browse/HBASE-7391
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Jean-Daniel Cryans
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 0.95.2
>
>
> From Ram in http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbo=
x/%3C00bc01cd31e6$7caf1320$760d3960$%25vasudevan@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.ed=
its
> for each of the region (I don=E2=80=99t have much data in the regions (~3=
00MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 =3D ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 =3D ~7GB.(Excluding actual data).  This seems to a very high
> initial memory foot print and this never allows me to split the logs and =
I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira