hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6581) Write to single replica in memory
Date Fri, 19 Sep 2014 22:49:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141483#comment-14141483

Andrew Wang commented on HDFS-6581:

bq. HDFS-4953 is not a subtask of HDFS-4949, and you yourself have said that it is an independent

Arpit, it's unfair to say that HDFS-4953 isn't related to HDFS-4949. ZCR makes the most sense
when it's used in combination with HDFS-4949 which is why we started working on ZCR in the
first place. We committed it to the HDFS-4949 branch first and brought it down to trunk before
the rest of the branch because other people were interested in using the new API.

bq. Also no numbers were cited or provided as evidence when calling the merge vote for HDFS-4949
so I don't think the community could reasonably be expected to take that into consideration.

I don't think this is a reasonable conclusion. HDFS-4953 is certainly related to HDFS-4949,
had benchmark results, and was already closely scrutinized by the community before the merge
(37 watchers, including yourself). It didn't seem necessary to bring it up the performance
again in the merge email, and we could have easily pointed at the existing results on HDFS-4953
if anyone asked for performance numbers.

I'm trying to be constructive here, which is why I provided a pointer to HDFS-6287. It's a
microbenchmark that can actually read at close to memory speed (GB/s), which isn't true of
a lot of other tests. Setting something up that writes and then reads using vecsum seems like
a reasonable benchmark. Not sure if you have libhdfs bindings for this work yet, but that'd
be the best thing if you want to adapt vecsum.

> Write to single replica in memory
> ---------------------------------
>                 Key: HDFS-6581
>                 URL: https://issues.apache.org/jira/browse/HDFS-6581
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, HDFS-6581.merge.03.patch,
HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch, HDFS-6581.merge.06.patch, HDFS-6581.merge.07.patch,
HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf
> Per discussion with the community on HDFS-5851, we will implement writing to a single
replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can revisit at a later

This message was sent by Atlassian JIRA

View raw message