hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on
Date Thu, 03 Nov 2016 11:45:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15632502#comment-15632502
] 

Rakesh R commented on HDFS-10885:
---------------------------------

bq. Yes, besides there is another reason that NN may not be ready when calling dfs.exists.

{code}
+      while (namesystem.isInSafeMode()) {
+        Thread.sleep(50);
+      }
{code}
Internally {{dfs.exists()}} is using {{FSNamesystem#getFileInfo()}} function, which will return
filestatus even if the NN is in safe mode. So the safemode checks is not required for {{#createMarkRunning()}}
, I think.

I just noticed that you have used {{DistributedFileSystem}} for creating the MOVER_PATH_ID
in StoragePolicySatisfier. I'd prefer not to use DFS approach as SPS daemon is part of Namenode.
{code}
+      DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(
+          FileSystem.getDefaultUri(conf), conf);
{code}

Alternate approach is to use FSNamesystem APIs directly like below. Again, StoragePolicySatisfier
is kept under BlockManager. BlockManager doesn't have FSNamesystem reference to call {{#getFileInfo()}}
and this function is not visible. [~umamaheswararao], [~drankye], any suggestions?

{code}
//1. StoragePolicySatisfier run()
     if (FSNamesystem.getFileInfo(MOVER_ID_PATH, true)) {
           // throw exception
     } else {
           // create MOVER_ID_PATH
     }

//2. StoragePolicySatisfier stop()
    // delete MOVER_ID_PATH

{code}


> [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-10885
>                 URL: https://issues.apache.org/jira/browse/HDFS-10885
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Wei Zhou
>            Assignee: Wei Zhou
>             Fix For: HDFS-10285
>
>         Attachments: HDFS-10800-HDFS-10885-00.patch, HDFS-10800-HDFS-10885-01.patch,
HDFS-10800-HDFS-10885-02.patch, HDFS-10885-HDFS-10285.03.patch, HDFS-10885-HDFS-10285.04.patch,
HDFS-10885-HDFS-10285.05.patch
>
>
> These two can not work at the same time to avoid conflicts and fight with each other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message