Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 23 May 2017 01:17:04 +0000 (UTC)
From: "Andrew Wang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13039369.1485918361000.268354.1495502224160@Atlassian.JIRA>
In-Reply-To: <JIRA.13039369.1485918361000@Atlassian.JIRA>
References: <JIRA.13039369.1485918361000@Atlassian.JIRA> <JIRA.13039369.1485918361881@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-11383) String duplication in
 org.apache.hadoop.fs.BlockLocation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 23 May 2017 01:17:08 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020515#comment-16020515 ] 

Andrew Wang commented on HDFS-11383:
------------------------------------

Hi Misha, could you provide some indicative comparison numbers for this change? I get that it's safe, but it'd be good to document the expected improvement in heap usage. This should be doable with a unit test if you can't revive your test cluster.

> String duplication in org.apache.hadoop.fs.BlockLocation
> --------------------------------------------------------
>
>                 Key: HDFS-11383
>                 URL: https://issues.apache.org/jira/browse/HDFS-11383
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Misha Dmitriev
>            Assignee: Misha Dmitriev
>         Attachments: HDFS-11383.01.patch
>
>
> I am working on Hive performance, investigating the problem of high memory pressure when (a) a table consists of a high number (thousands) of partitions and (b) multiple queries run against it concurrently. It turns out that a lot of memory is wasted due to data duplication. One source of duplicate strings is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds, topologyPaths, hosts, names, may collectively use up to 6% of memory in my benchmark, causing (together with other problematic classes) a huge memory spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are wasted due to duplication.
> I think we need to add calls to String.intern() in the BlockLocation constructor, like:
> {code}
> this.hosts = internStringsInArray(hosts);
> ...
> private void internStringsInArray(String[] sar) {
>   for (int i = 0; i < sar.length; i++) {
>     sar[i] = sar[i].intern();
>   }
> }
> {code}
> String.intern() performs very well starting from JDK 7. I've found some articles explaining the progress that was made by the HotSpot JVM developers in this area, verified that with benchmarks myself, and finally added quite a bit of interning to one of the Cloudera products without any issues.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org