hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Saputra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5851) Support memory as a storage medium
Date Fri, 27 Jun 2014 18:44:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046247#comment-14046247

Henry Saputra commented on HDFS-5851:

HI [~sanjay.radia], I was looking at the JIRA and proposal and I have some questions related
to it:
1. I did not see where the memory will be allocated for the DDM proposal. Is it similar to
HDFS-4949 to use the memory from Datanode?
2. As for the APIs, would it be new Hadoop FS (Java) APIs or higher level construct to store
data in memory because it seemed that the proposal only relying on file path to indicate trying
to use in-memory cache
3. In the problem statement of the proposal seemed like there would be policy to manage how
data should be store in  memory per application but I could not find details about how to
achieve it. Some applications may need to have quick access to some small portion of data
more significant (eg: newer time series data) whereas some others may be need to store more
(eg: large Hive query)
4. In term of discardability, what is the "eviction" policy for such data and how control
or fine tune it if needed.

Maybe it was discussed in the in-person happened before but I could not find it in the meet
Thanks for driving this new feature.

> Support memory as a storage medium
> ----------------------------------
>                 Key: HDFS-5851
>                 URL: https://issues.apache.org/jira/browse/HDFS-5851
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf,
SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf
> Memory can be used as a storage medium for smaller/transient files for fast write throughput.
> More information/design will be added later.

This message was sent by Atlassian JIRA

View raw message