hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arpit Agarwal <aagar...@hortonworks.com>
Subject [VOTE] Merge HDFS-6581 to trunk - Writing to replicas in memory.
Date Tue, 23 Sep 2014 00:24:40 GMT
HDFS Devs,

We propose merging the HDFS-6581 development branch to trunk.

The work adds support to write to HDFS blocks in memory. The target use
case covers applications writing relatively small, intermediate data sets
with low latency. We introduce a new CreateFlag for the existing CreateFile
API. HDFS will subsequently attempt to place replicas of file blocks in
local memory with disk writes occurring off the hot path. The current
design is a simplification of original ideas from Sanjay Radia on HDFS-5851.

Key goals of the feature were minimal API changes to reduce application
burden and best effort data durability. The feature is optional and
requires appropriate DN configuration from administrators.

Design doc:
https://issues.apache.org/jira/secure/attachment/12661926/HDFSWriteableReplicasInMemory.pdf

Test plan:
https://issues.apache.org/jira/secure/attachment/12669452/Test-Plan-for-HDFS-6581-Memory-Storage.pdf

There are 28 resolved sub-tasks under HDFS-6581, 3 open tasks for
tests+Jenkins issues  and 7 open subtasks tracking planned improvements.
The latest merge patch is 3300 lines of changed code of which 1300 lines is
new and updated tests. Merging the branch to trunk will allow HDFS
applications to start evaluating the feature. We will continue work on
documentation, performance tuning and metrics in parallel with the vote and
post-merge.

Contributors to design and code include Xiaoyu Yao, Sanjay Radia, Jitendra
Pandey, Tassapol Athiapinya, Gopal V, Bikas Saha, Vikram Dixit, Suresh
Srinivas and Chris Nauroth.

Thanks to Haohui Mai, Colin Patrick McCabe, Andrew Wang, Todd Lipcon, Eric
Baldeschwieler and Vinayakumar B for providing useful feedback on
HDFS-6581, HDFS-5851 and sub-tasks.

The vote runs for the usual 7 days and will expire at 12am PDT on Sep 30.
Here is my +1 for the merge.

Regards,
Arpit

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message