hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
Date Wed, 19 Apr 2017 20:27:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975451#comment-15975451

Andrew Wang commented on HDFS-11643:

Hi Sammi, took a deeper look at this, mostly just nitty and API stuff:

* We can simplify and rename the setter {{enforceAsReplicatedFile(Boolean)}} to just be {{replicate()}}
with no parameters, since the default is false. I think this could also just be a primitive
boolean, since the {{dfs.create}} takes a primitive boolean rather than a nullable Boolean.
* asReplicatedFile, recommend we rename to {{shouldReplicate}} and make the getter protected
* We also should add some Javadoc to {{replicateFile}} and {{setEcPolicyName}} to explain
what they do and that they are exclusive.

I'd prefer to be more consistent about {{ecPolicyName}} and {{asReplicated}} being exclusive
parameters and throwing an exception. We fixed the builder, but a few other places I noticed:
* ClientProtocol: docs say that ecPolicyName is ignored, instead, let's make it clear that
it's invalid to set {{asReplicated}} to true and also a non-null {{ecPolicyName}}.
* Builder#build: Please add some additional help text for the user when the Precondition check
* FSNamesystem#startFileInt: Let's check exclusivity here as well, can do it outside the write
lock where we do the invalid path check.

* Note that the DiskBalancer is different from the Balancer. We should be adding a test to
TestBalancer instead.

Recommend reformatting these chained calls in TestErasureCodingPolicies like this for clarity:



> Balancer fencing fails when writing erasure coded lock file
> -----------------------------------------------------------
>                 Key: HDFS-11643
>                 URL: https://issues.apache.org/jira/browse/HDFS-11643
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover, erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
>            Assignee: SammiChen
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, HDFS-11643.003.patch
> At startup, the balancer writes its hostname to the lock file and calls hflush(). hflush
is not supported for EC files, so this fails when the entire filesystem is erasure coded.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message