hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6673) Add Delimited format supports for PB OIV tool
Date Sat, 24 Jan 2015 00:57:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290294#comment-14290294

Andrew Wang commented on HDFS-6673:

bq. The test is on a 3G fsimage which can easily fit in the working set of your laptop...Multiple
users are running much larger clusters, where their fsimage can be as big as 40G

I know there are larger customers out there, but as I said above, we surveyed the sizes of
our customer's fsimages, and this was one of the larger ones. Thus this solution will work
for most production deployments. Also, even a 40GB image easily fits in memory on server these

bq. but given we have HDFS-6293 as a solid solution today, I would much rather to see this
tool to be capable of handling fsimages from the real, large-scale production runs

HDFS-6293 doesn't include metadata newer than the old format. It also requires the NN to write
out an additional old fsimage just for OIV alongside the real one. These seem like pretty
big drawbacks to me, and are addressed by this tool. I think calling it "half-baked" is unfair
considering it provides greater functionality.

Anyway, thanks for not -1'ing, I'll commit this now. Let's continue this discussion in a follow-on

> Add Delimited format supports for PB OIV tool
> ---------------------------------------------
>                 Key: HDFS-6673
>                 URL: https://issues.apache.org/jira/browse/HDFS-6673
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: 2.4.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>         Attachments: HDFS-6673.000.patch, HDFS-6673.001.patch, HDFS-6673.002.patch, HDFS-6673.003.patch,
HDFS-6673.004.patch, HDFS-6673.005.patch
> The new oiv tool, which is designed for Protobuf fsimage, lacks a few features supported
in the old {{oiv}} tool. 
> This task adds supports of _Delimited_ processor to the oiv tool. 

This message was sent by Atlassian JIRA

View raw message