hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6581) Write to single replica in memory
Date Wed, 17 Sep 2014 23:51:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138223#comment-14138223

Arpit Agarwal commented on HDFS-6581:

Thanks for the note. We have been busy getting the basic feature in and stabilizing the branch.
I intend to call a merge vote for the branch later this week. Most of the things you mention
are somewhat open ended tasks and we plan to address post-merge:
# Benchmarking writes to memory. This might suggest further work to improve the data write
# Benchmark the impact of CRC computation, evaluate moving it off the hot path.
# Improve eviction, there's multiple ideas floating around, including integration with CCM.

Contributions would be welcome on any of these.

bq. It would be nice to see that would could get a substantial fraction of memory bandwidth
when writing to a single replica in-memory.
The comparison will be interesting but I can tell you without measurement it is not going
to be a substantial fraction of memory bandwidth. We are still going through DataTransferProtocol
with all the copies and overhead that involves.

> Write to single replica in memory
> ---------------------------------
>                 Key: HDFS-6581
>                 URL: https://issues.apache.org/jira/browse/HDFS-6581
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, HDFS-6581.merge.03.patch,
HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf
> Per discussion with the community on HDFS-5851, we will implement writing to a single
replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can revisit at a later

This message was sent by Atlassian JIRA

View raw message