Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Date: Fri, 22 Aug 2014 09:30:11 +0000 (UTC)
From: "Hao Chen (JIRA)" <jira@apache.org>
To: hdfs-dev@hadoop.apache.org
Message-ID: <JIRA.12736024.1408699808143.11177.1408699811248@arcas>
In-Reply-To: <JIRA.12736024.1408699808143@arcas>
References: <JIRA.12736024.1408699808143@arcas>
Subject: [jira] [Created] (HDFS-6914) Resolve huge memory consumption Issue
 with OIV processing PB-based fsimages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Hao Chen created HDFS-6914:
------------------------------

             Summary: Resolve huge memory consumption Issue with OIV processing PB-based fsimages
                 Key: HDFS-6914
                 URL: https://issues.apache.org/jira/browse/HDFS-6914
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.4.1
            Reporter: Hao Chen
             Fix For: 2.5.1


For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. 

However  due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server.

To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%.


--
This message was sent by Atlassian JIRA
(v6.2#6252)