hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1702) Reduce buffer copies when data is written to DFS
Date Thu, 08 May 2008 18:16:48 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595354#action_12595354
] 

Raghu Angadi commented on HADOOP-1702:
--------------------------------------

Thanks for the review Hairong.

- #1 :  interesting suggestion. It would be a change in protocol that affects other datanode
transfers like reading etc. We rarely send partial packets (mainly fsync). Also checksum data
is less than one percent of the packet. I hope it is ok for this patch. Note that this is
not a memory copy that did not exist before (previously all the data was copied).

- #2: Yes it is a change from previous behavior. Before this patch it didn't matter since
we handled 512 bytes at a time. The receiving datanode verifies the checksum anyway. Checking
checksum after tunneling data downstream (theoretically) reduces latency. This is the same
reason datanode first sends the data to mirror and then stores the data locally.

- #3. sure.
- #4. yes.

> Reduce buffer copies when data is written to DFS
> ------------------------------------------------
>
>                 Key: HADOOP-1702
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1702
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>             Fix For: 0.18.0
>
>         Attachments: HADOOP-1702.patch, HADOOP-1702.patch, HADOOP-1702.patch, HADOOP-1702.patch,
HADOOP-1702.patch, HADOOP-1702.patch
>
>
> HADOOP-1649 adds extra buffering to improve write performance.  The following diagram
shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of
our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.
> {noformat}
>        (1)                 (2)          (3)                 (5)
>    +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to
Mirror  
>    | (buffer)                  (socket)           |  (4)
>    |                                              +--||--+
>  =====                                                    |
>  =====                                                  =====
>  (disk)                                                 =====
> {noformat}
> Currently loops that read and write block data, handle one checksum chunk at a time.
By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5). 
> Similarly some copies can be reduced when clients read data from the DFS.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message