Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Mon, 5 May 2014 11:19:16 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12710933.1398695378984.229320.1399288756099@arcas>
In-Reply-To: <JIRA.12710933.1398695378984@arcas>
References: <JIRA.12710933.1398695378984@arcas>
Subject: [jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based
 fsimages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989427#comment-13989427 ] 

Hadoop QA commented on HDFS-6293:
---------------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12643332/HDFS-6293.000.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6812//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6812//console

This message is automatically generated.

> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
>                 Key: HDFS-6293
>                 URL: https://issues.apache.org/jira/browse/HDFS-6293
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Kihwal Lee
>            Assignee: Haohui Mai
>            Priority: Blocker
>         Attachments: HDFS-6293.000.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf. 
> Due to the internal layout changes introduced by the protobuf-based fsimage, OIV consumes excessive amount of memory.  We have tested with a fsimage with about 140M files/directories. The peak heap usage when processing this image in pre-protobuf (i.e. pre-2.4.0) format was about 350MB.  After converting the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of heap (max new size was 1GB).  It should be possible to process any image with the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.  I also noticed that the secret manager section has no tokens while there were unexpired tokens in the original image (pre-2.4.0).  I did not check whether they were also missing in the new pb fsimage.


--
This message was sent by Atlassian JIRA
(v6.2#6252)