hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Kace (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10629) Federation Router
Date Tue, 04 Oct 2016 01:05:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543962#comment-15543962
] 

Jason Kace commented on HDFS-10629:
-----------------------------------

??Should rename method do some validation to make sure src ns != destination ns???
\\ \\
* The rename operation currently does an inner join of the potential nameservices for the
destination and source.  If none match, the request is failed.  Once the source file is located
(may be in 1-N different nameservices), at least one of the destination paths must be in the
same nameservice.  If these conditions are not met, the rename fails.  The rename can only
change the metadata on a single NN, it does not move data between subclusters.  The moving
data role is reserved for the rebalancer.

??Router#startTime and Router#routerId not used. Let us add them later when they are required.
There are other unused methods in other files. Can you check if they are needed?  It seems
abandonBlock just needs to refer to one name service based on the block id, thus can call
invokeSingle.  There are still other javadoc and code mismatch in several file??
\\ \\
* I will address these in the next patch.

??failIfLocked isn't used. What is for???
\\ \\
* During rebalancing there is a synchronization window where none of the routers can allow
metadata changes to occur, otherwise data may be lost.  The lock is intended to allow read-operations
and fail write-operations during this window.  It is not part of this patch and I will remove
it.

??How about dynamic stats such as numOfFiles which are only used by rebalancer???
\\ \\
* For now we only need the IDs and the IP addresses.  I can remove the other fields.

??Other APIs such as mkdir were modfiied to check all locations to ensure the directory doesn't
exist before permitting a create.  Some test case to cover multiple locations will be useful.??
\\ \\
* I have some tests for mutiple locations, will add them to the next patch.

??Wonder why mkdirs check all locations during validation, while only use the first location
for the actual mkdirs call.??
\\ \\
* This is one of the core features of the FileSubclusterLocator, which is not part of this
patch.  The design currently works in this manner:
*# Check the mount table for the path
*# If there is only one destination, use it
*# If there are more than one destinations, build a prioritized list and try them in order
to limit excess NN traffic.  The highest priority is determined by a consistent hash algorithm
on the path.
\\ \\
* When executing mkdirs, it is important to check that the dir doesn't already exist in any
namespace.  In an ideal world the directory would always be in the highest priority location,
but there are corner cases where that isn't already true.

* When creating all files, including directories, the highest priority location/nameservice
(via consistent hashing) is used.  This helps ensure a hit on the first attempt when looking
for the file.  All create/mkdir operations use the highest priority path.  If a rename subsequently
happens the file may not be in the highest priority location, something the rebalancer can
consider.

??It will be useful to document the scenarios for multiple destination, what specific mount
table settings this will cover, nfly, mergeFs, etc.??
\\ \\
* The current design will support 1-N different destinations for a mount point, with each
destination having a 1:1 map to an actual location (nameservice + path).  A mount can map
to multiple paths within the same nameservice, multiple nameservices or both.

??It might be useful to check PathLocation#hasMultipleDestinations first. If there is only
one destination, getFileInfo can be skipped.??
\\ \\
* Good point, there are some cases where checks can be skipped.

> Federation Router
> -----------------
>
>                 Key: HDFS-10629
>                 URL: https://issues.apache.org/jira/browse/HDFS-10629
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs
>            Reporter: Inigo Goiri
>            Assignee: Jason Kace
>         Attachments: HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch,
HDFS-10629-HDFS-10467-004.patch, HDFS-10629.000.patch, HDFS-10629.001.patch
>
>
> Component that routes calls from the clients to the right Namespace. It implements {{ClientProtocol}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message