hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
Date Fri, 06 Jan 2017 22:39:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806015#comment-15806015
] 

Hadoop QA commented on HDFS-6914:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  0s{color} | {color:blue}
Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} | {color:red}
HDFS-6914 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-6914 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12671201/HDFS-6914.v2.patch
|
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/18082/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-6914
>                 URL: https://issues.apache.org/jira/browse/HDFS-6914
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.4.1
>            Reporter: Hao Chen
>              Labels: hdfs
>         Attachments: HDFS-6914.patch, HDFS-6914.v2.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in production, we internally
need to automatically export fsimage to delimited text files in LSR style and then analyse
with hive or pig or build system metrics for real-time analyzing. 
> However  due to the internal layout changes introduced by the protobuf-based fsimage,
OIV processing program consumes excessive amount of memory. For example, in order to export
the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable
and impacted performance of other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory consumption of
OIV LSR processing by 50%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message