hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3164) Use FileChannel.transferTo() when data is read from DataNode.
Date Mon, 21 Apr 2008 19:47:21 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Raghu Angadi updated HADOOP-3164:
---------------------------------

    Attachment: HADOOP-3164.patch

Updated patch includes feedback from Konstantin. Also improves documentation.

Regd Sam's experiment on Mac OS :
8x is very surprising. Sam sent me iostat during the test and the main difference seems to
be that size of each read from the disk with out the patch is around 350 KB and just 4 KB
with the patch.  The latest patch uses real socket channel with transferTo() on all the platforms.
Please check if it improves the test.

> Use FileChannel.transferTo() when data is read from DataNode.
> -------------------------------------------------------------
>
>                 Key: HADOOP-3164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3164
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>             Fix For: 0.18.0
>
>         Attachments: HADOOP-3164.patch, HADOOP-3164.patch, HADOOP-3164.patch, HADOOP-3164.patch,
HADOOP-3164.patch
>
>
> HADOOP-2312 talks about using FileChannel's [{{transferTo()}}|http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transferTo(long,%20long,%20java.nio.channels.WritableByteChannel)]
and [{{transferFrom()}}|http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transferFrom(java.nio.channels.ReadableByteChannel,%20long,%20long)]
in DataNode. 
> At the time DataNode neither used NIO sockets nor wrote large chunks of contiguous block
data to socket. Hadoop 0.17 does both when data is seved to clients (and other datanodes).
I am planning to try using transferTo() in the trunk. This might reduce DataNode's cpu by
another 50% or more.
> Once HADOOP-1702 is committed, we can look into using transferFrom().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message