hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5851) Support memory as a storage medium
Date Sun, 29 Jun 2014 02:32:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047035#comment-14047035

Konstantin Shvachko commented on HDFS-5851:

Few questions as I don't see it covered in the design document:
# Heterogeneous storage is implemented but not enabled, as for now it can only allocate the
StorageType.DEFAULT blocks. This seems to be the first extension to other StorageTypes. Why
memory type is getting prioritized ahead of say SSDs?
# I understand the design assumes that only one DN will hold a memory replica of a  DDM block.
This will increase the latency accessing that block for a single client, but it also makes
this DN a bottleneck for many clients trying to access the same data.
# If I understood correctly the proposal is to not introduce new APIs for discarding unnecessary
or lost data, but handle it using a discardability policy. What is the policy?
#- In this regard Eric's idea of ZK ephemeral nodes is interesting, but probably not directly
applicable, as a file should not be discarded only because its creator quit.
# Eviction policy is another thing which needs clarification.
# What do you mean by "static allocation of memory"? A configuration parameter for DNs?

> Support memory as a storage medium
> ----------------------------------
>                 Key: HDFS-5851
>                 URL: https://issues.apache.org/jira/browse/HDFS-5851
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf,
SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf
> Memory can be used as a storage medium for smaller/transient files for fast write throughput.
> More information/design will be added later.

This message was sent by Atlassian JIRA

View raw message