hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9763) Add merge api
Date Tue, 01 Mar 2016 19:12:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174219#comment-15174219

Jitendra Nath Pandey commented on HDFS-9763:

 I don't think this API will address TOCTOU because hive will need multiple merge calls anyway
for a given query, to reach final stage. In the absence of transactional support in the file
system, as suggested by Haohui, it is difficult to support such semantics at HDFS level.

  However, I do see the need of hive to avoid hundreds or thousands of individual rename operations.
This also seems to be a pretty general case for many map-reduce based data ingest jobs that
write data all over the place and finally need to consolidate in a  final directory, after
all the related jobs are successful.
  IMO, the requirement here is of a 'rename' operation that merges the directories instead
of overwriting, or throwing already-exists-exception. However, it can be argued that its better
to add a new 'merge' API, instead of overloading the 'rename'. 
  I see that there are two main design points here that need to be carefully thought through
and agreed upon
1) How do we resolve conflicts in file names
   I think the proposal handles it elegantly by specifying a policy. In fact I would love
to change our 'rename' to support different policies that also provides merge capability,
but I can accept a separate API for compatibility and simplicity.
2) The O(N) problem.
   It is not so bad because its not recursive. There is no scope of recursion here.
   Still if a directory has a lot of files, an iterative approach is feasible here, because
the source directory will get smaller after every iteration. We do have precedence of iteration
for example 'listStatus'. This will avoid the complexity in the NN to release the lock in

> Add merge api
> -------------
>                 Key: HDFS-9763
>                 URL: https://issues.apache.org/jira/browse/HDFS-9763
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Ashutosh Chauhan
>            Assignee: Xiaobing Zhou
>         Attachments: HDFS_Merge_API_Proposal.pdf
> It will be good to add merge(Path dir1, Path dir2, ... ) api to HDFS. Semantics will
be to move all files under dir1 to dir2 and doing a rename of files in case of collisions.
> In absence of this api, Hive[1] has to check for collision for each file and then come
up unique name and try again and so on. This is inefficient in multiple ways:
> 1) It generates huge number of calls on NN (atleast 2*number of source files in dir1)
> 2) It suffers from TOCTOU[2] bug for client picked up name in case of collision.
> 3) Whole operation is not atomic.
> A merge api outlined as above will be immensely useful for Hive and potentially to other
HDFS users.
> [1] https://github.com/apache/hive/blob/release-2.0.0-rc1/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2576
> [2] https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use

This message was sent by Atlassian JIRA

View raw message