hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6581) Write to single replica in memory
Date Thu, 18 Sep 2014 01:52:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138383#comment-14138383

Colin Patrick McCabe commented on HDFS-6581:

bq. I don't recall any performance numbers were published before merging to trunk. Looking
through the Jira even now I don't see any numbers. Maybe it's there - feel free to give a

There are a bunch of performance numbers on HDFS-4953 (a subtask of HDFS-4949).  Enabling
zero-copy reads was an important goal of HDFS-4949.  On HDFS-4953, Todd and I discussed how
to get close to the optimal 9GB/s per core.  More generally, we knew the HDFS read path could
push a few gigs a second even without zero-copy because we'd done the testing beforehand (some
of the results are on HDFS-347, some of that discussion is scattered other places).  So we
were confident that HDFS-4949 was a performance improvement at the time of the merge.

bq. The current direction is based on broad discussion with the community here and on HDFS-5851,
and the overall consensus was to keep DataTransferProtocol in the picture for now.

I'm fine with keeping {{DataTransferProtocol}} in the picture for now.  But you should still
be able to show a performance improvement when reading (not writing) the data which you have
previously written to the single replica in memory.  If there are flaws in this (for example,
if short-circuit doesn't yet work for these blocks, causing slowness) then we should address
this before merging.

We should also quantify exactly what the write performance is, even if it's not as good as
it could be.  This will tell us what we need to work on in the future and what the priorities
should be.

> Write to single replica in memory
> ---------------------------------
>                 Key: HDFS-6581
>                 URL: https://issues.apache.org/jira/browse/HDFS-6581
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, HDFS-6581.merge.03.patch,
HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf
> Per discussion with the community on HDFS-5851, we will implement writing to a single
replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can revisit at a later

This message was sent by Atlassian JIRA

View raw message