hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kannan Rajah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11905) Abstraction for LocalDirAllocator
Date Fri, 08 May 2015 23:47:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535930#comment-14535930

Kannan Rajah commented on HADOOP-11905:

[~cmccabe] The goal is not to make changes to DataNode or anything in HDFS layer. It is to
provide an alternative for shuffle. In Apache, this is done using HTTP. In a file system like
MapR, the map outputs can be written to DFS instead using the notion of MapR local volumes.
While I was trying to see how we can unify the 2 approaches, I realized that the fundamental
assumption is that local disks can only be managed by local file system. If we remove this
assumption and provide the new abstraction to hadoop common, then we can change Tez app master
(for example) to use it. The net effect is that by turning on a setting (yarn.local.disk.path.allocator.use-dfs
= true), we can make it use DFS based shuffle instead of HTTP based shuffle. I already have
a working Tez App Master using this abstraction. The changes are trivial and limited to about
20 files. I was going to file a Tez JIRA once this got accepted upstream.

2 changes needed top Tez classes:
LocalDirAllocator allocator = new LocalDirAllocator()
LocalDiskPathAllocator allocator = LocalDiskUtil.getPathAllocator()

RawLocalFileSystem fs = new FileSystem.getLocal()
FileSystem fs = LocalDiskUtil.getFileSystem()

> Abstraction for LocalDirAllocator
> ---------------------------------
>                 Key: HADOOP-11905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11905
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.5.2
>            Reporter: Kannan Rajah
>            Assignee: Kannan Rajah
>              Labels: BB2015-05-TBR
>             Fix For: 2.7.1
>         Attachments: 0001-Abstraction-for-local-disk-path-allocation.patch
> There are 2 abstractions used to write data to local disk.
> LocalDirAllocator: Allocate paths from a set of configured local directories.
> LocalFileSystem/RawLocalFileSystem: Read/write using java.io.* and java.nio.*
> In the current implementation, local disk is managed by guest OS and not HDFS. The proposal
is to provide a new abstraction that encapsulates the above 2 abstractions and hides who manages
the local disks. This enables us to provide an alternate implementation where a DFS can manage
the local disks and it can be accessed using HDFS APIs. This means the DFS maintains a namespace
for node local directories and can create paths that are guaranteed to be present on a specific
> Here is an example use case for Shuffle: When a mapper writes intermediate data using
this new implementation, it will continue write to local disk. When a reducer needs to access
data from a remote node, it can use HDFS APIs with a path that points to that node’s local
namespace instead of having to use HTTP server to transfer the data across nodes.
> New Abstractions
> 1. LocalDiskPathAllocator
> Interface to get file/directory paths from the local disk namespace.
> This contains all the APIs that are currently supported by LocalDirAllocator. So we just
need to change LocalDirAllocator to implement this new interface.
> 2. LocalDiskUtil
> Helper class to get a handle to LocalDiskPathAllocator and the FileSystem
> that is used to manage those paths.
> By default, it will return LocalDirAllocator and LocalFileSystem.
> A supporting DFS can return DFSLocalDirAllocator and an instance of DFS.
> 3. DFSLocalDirAllocator
> This is a generic implementation. An allocator is created for a specific node. It uses
Configuration object to get user configured base directory and appends the node hostname to
it. Hence the returned paths are within the node local namespace.

This message was sent by Atlassian JIRA

View raw message