hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5851) Support memory as a storage medium
Date Tue, 20 May 2014 21:54:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004031#comment-14004031
] 

Arpit Agarwal commented on HDFS-5851:
-------------------------------------

Minutes from Google Hangout:

Wrt to the mechanism to support memory caching there was high level agreement on the implementation
phases roughly as:
* 1st phase - streaming socket write, but mlock on DN side so that it keeps it for readers.
** Make this work for a single replica
** Separately (in another Jira) investigate write-pipeline improvements because the write-pipeline
has not been optimized. This should give us some initial performance numbers and one can start
using this mechanism. [~tlipcon] (?) has a prototype.

* 2nd phase - Explore short-circuit write, but datanode still mlocks. We had a quick discussion
on short-circuit write being tricky
** Recovery issues (RBW)
** Client can do things that can get the DN confused (e.g. truncate/append the file after
close)

* Future phases
** Add lazy replication to other replicas (note earlier phases allowed only 1 replica)
** Direct writes to memory by memory-mapping the file

Discussion on discardability:
* Shouldn't this be property of file (such a replica count of 1) rather than a a property
of /.reserved/ddm?
** This needs further discussion on the jira.
* Why the two layer approach?
** We don't want to necessarily put load on NN for intermediate files and hence the 2nd layer.

> Support memory as a storage medium
> ----------------------------------
>
>                 Key: HDFS-5851
>                 URL: https://issues.apache.org/jira/browse/HDFS-5851
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf,
SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf
>
>
> Memory can be used as a storage medium for smaller/transient files for fast write throughput.
> More information/design will be added later.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message