hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
Date Fri, 11 Jan 2019 20:48:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740771#comment-16740771
] 

Wei-Chiu Chuang commented on HDFS-14061:
----------------------------------------

Nit: better to use parameterized log format in slf4j. I see that you have HDFS-14125 opened
to address though.
{code}
System.out.println("Warning: The cluster setup does not support " +
{code}
Use System.err instead of System.out

{code}
final DatanodeInfo[] report = dfs.getClient().getNamenode()
        .getDatanodeReport(HdfsConstants.DatanodeReportType.ALL);
{code}
So we're assuming we use all DataNodes in the cluster to verify if the EC policy can be applied,
regardless of whether they are dead/decomm'ed/in maintenance. Probably makes sense. The code
was inherited from HDFS-12946, and I didn't check if this is considered. I'll just leave the
note here. 

Ideally getPolicy() should not return a null if policy name is not found. It won't break the
functionality here though because the policy is enabled prior to that. So the policy is guaranteed
to exist (unless a race condition where admin removes the policy concurrently)

Test: FYI {{GenericTestUtils#SystemErrCapturer}} can be used for checking syserr output. Something
for you to consider next time.

> Check if the cluster topology supports the EC policy before setting, enabling or adding
it
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-14061
>                 URL: https://issues.apache.org/jira/browse/HDFS-14061
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding, hdfs
>    Affects Versions: 3.1.1
>            Reporter: Kitti Nanasi
>            Assignee: Kitti Nanasi
>            Priority: Major
>         Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, HDFS-14061.003.patch,
HDFS-14061.004.patch
>
>
> HDFS-12946 introduces a command for verifying if there are enough racks and datanodes
for the enabled erasure coding policies.
> This verification could be executed for the erasure coding policy before enabling, setting
or adding it and a warning message could be written if the verify fails, or the policy setting
could be even failed in this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message