hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
Date Tue, 15 Aug 2017 18:00:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127645#comment-16127645

Andrew Wang commented on HDFS-10285:

Hi Uma,

bq. Now quick question on your example above “a user calling the shell command and polling
until it's done” , you mean command should blocked by polling internally? or user will call
status check periodically? How much time server should hold the status?

This would be a user periodically asking for status. From what I know of async API design,
callbacks are preferred over polling since it solves the question about how long the server
needs to hold the status.

I'd be open to any proposal here, I just think the current "isSpsRunning" API is insufficient.
Did you end up filing a ticket to track this?

bq. <C-DN cases>

If I were to paraphrase, the NN is the ultimate arbiter, and the operations being performed
by C-DNs are idempotent, so duplicate work gets dropped safely. I think this still makes it
harder to reason about from a debugging POV, particularly if we want to extend this to something
like EC conversion that might not be idempotent.

bq. Mainly our motivation to offload work was mainly to avoid tracking at block level results.
Now NN just tracks file level results. C-DN track all block level movements and send the result
back. We also thinking to use this kind of model for converting regular files to EC and also
HDFS-12090 is one of the use case they wanted to use it. I think keeping all such monitoring
logic into NN will be definitely overhead on NN. My feeling is, we should think to offload
work as much as possible from NN.

I like the idea of offloading work in the abstract, but I don't know how much work we really
offload in this situation. The NN still needs to track everything at the file level, which
is the same order of magnitude as the block level. The NN is still doing blockmanagement and
processing IBRs for the block movement. Distributing tracking work to the C-DNs adds latency
and makes the system more complicated.

> Storage Policy Satisfier in Namenode
> ------------------------------------
>                 Key: HDFS-10285
>                 URL: https://issues.apache.org/jira/browse/HDFS-10285
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HDFS-10285-consolidated-merge-patch-00.patch, HDFS-10285-consolidated-merge-patch-01.patch,
HDFS-SPS-TestReport-20170708.pdf, Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, Storage-Policy-Satisfier-in-HDFS-May10.pdf
> Heterogeneous storage in HDFS introduced the concept of storage policy. These policies
can be set on directory/file to specify the user preference, where to store the physical block.
When user set the storage policy before writing data, then the blocks could take advantage
of storage policy preferences and stores physical block accordingly. 
> If user set the storage policy after writing and completing the file, then the blocks
would have been written with default storage policy (nothing but DISK). User has to run the
‘Mover tool’ explicitly by specifying all such file names as a list. In some distributed
system scenarios (ex: HBase) it would be difficult to collect all the files and run the tool
as different nodes can write files separately and file can have different paths.
> Another scenarios is, when user rename the files from one effected storage policy file
(inherited policy from parent directory) to another storage policy effected directory, it
will not copy inherited storage policy from source. So it will take effect from destination
file/dir parent storage policy. This rename operation is just a metadata change in Namenode.
The physical blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult for admins from
distributed nodes(ex: region servers) and running the Mover tool. 
> Here the proposal is to provide an API from Namenode itself for trigger the storage policy
satisfaction. A Daemon thread inside Namenode should track such calls and process to DN as
movement commands. 
> Will post the detailed design thoughts document soon. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message