hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
Date Wed, 19 Apr 2017 20:27:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975451#comment-15975451
] 

Andrew Wang commented on HDFS-11643:
------------------------------------

Hi Sammi, took a deeper look at this, mostly just nitty and API stuff:

HdfsDataOutputStreamBuilder:
* We can simplify and rename the setter {{enforceAsReplicatedFile(Boolean)}} to just be {{replicate()}}
with no parameters, since the default is false. I think this could also just be a primitive
boolean, since the {{dfs.create}} takes a primitive boolean rather than a nullable Boolean.
* asReplicatedFile, recommend we rename to {{shouldReplicate}} and make the getter protected
* We also should add some Javadoc to {{replicateFile}} and {{setEcPolicyName}} to explain
what they do and that they are exclusive.

I'd prefer to be more consistent about {{ecPolicyName}} and {{asReplicated}} being exclusive
parameters and throwing an exception. We fixed the builder, but a few other places I noticed:
* ClientProtocol: docs say that ecPolicyName is ignored, instead, let's make it clear that
it's invalid to set {{asReplicated}} to true and also a non-null {{ecPolicyName}}.
* Builder#build: Please add some additional help text for the user when the Precondition check
fires
* FSNamesystem#startFileInt: Let's check exclusivity here as well, can do it outside the write
lock where we do the invalid path check.

Tests:
* Note that the DiskBalancer is different from the Balancer. We should be adding a test to
TestBalancer instead.

Recommend reformatting these chained calls in TestErasureCodingPolicies like this for clarity:

{noformat}
    fs.newFSDataOutputStreamBuilder(filePath).setEcPolicyName(ecPolicyName)
        .enforceAsReplicatedFile(false).build().close();
{noformat}

{noformat}
    fs.newFSDataOutputStreamBuilder(filePath)
        .setEcPolicyName(ecPolicyName)
        .enforceAsReplicatedFile(false)
        .build()
        .close();
{noformat}

> Balancer fencing fails when writing erasure coded lock file
> -----------------------------------------------------------
>
>                 Key: HDFS-11643
>                 URL: https://issues.apache.org/jira/browse/HDFS-11643
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover, erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
>            Assignee: SammiChen
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, HDFS-11643.003.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls hflush(). hflush
is not supported for EC files, so this fails when the entire filesystem is erasure coded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message