hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
Date Wed, 26 Nov 2014 05:35:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225783#comment-14225783
] 

Vinayakumar B commented on HDFS-7310:
-------------------------------------

Thanks [~umamaheswararao] for the review.
bq. Seems like now when you plan to move block across storage in a DN and if it is not on
transient storage, you are not doing checksum calculation. my point is, we may need to compute
checksum for having data integrity when moving across storage? For some reason, that is not
necessary, please let me know and also in such case we need to change below log though
On transient storage, checksum will not be available. So while moving a block from transient
storage to other storages checksum needs to be calculated during lazyPersist.
But in this case, moving from persistent to persistent, we can just copy the already existing
checksum. Later if any checksum failures, then it will be detected during scans.
I will update the log message.

> Mover can give first priority to local DN if it has target storage type available in
local DN
> ---------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7310
>                 URL: https://issues.apache.org/jira/browse/HDFS-7310
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover
>    Affects Versions: 3.0.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Vinayakumar B
>         Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch
>
>
> Currently Mover logic may move blocks to any DN which had target storage type. But if
the src DN has target storage type then mover can give highest priority to local DN. If local
DN does not contains target storage type, then it can assign to any DN as the current logic
does.
>   This is a thought, have not go through the code fully yet.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message