hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trevor Robinson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
Date Mon, 29 Oct 2012 06:57:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485869#comment-13485869
] 

Trevor Robinson commented on HDFS-3529:
---------------------------------------

The two Jenkins failures seem to be the same issue:

# One {{BlockReceiver$PacketResponder}} thread is interrupted in flushing an ack to an upstream
datanode.
# Another {{BlockReceiver$PacketResponder}} thread gets an {{EOFException}} trying to read
the length prefix of an ack.
# An NPE occurs during {{PacketReceiver.mirrorPacketTo}} (called via {{DataXceiver.writeBlock}})
because the packet buffer is null, which should only happen if {{close}} has been called.

I'm not sure which of the three caused the others, or if there's some other cause. I ran hundreds
of iterations of {{TestLoadGenerator}} in a loop, and cannot reproduce a failure locally.
I'd appreciate any suggestions on how to go about reproducing it. For example, is it reasonable
to attach a patch with extra logging, or does that disrupt builds for other issues? Or is
it possible to get access to a machine with the same configuration?
                
> Use direct buffers for data in write path
> -----------------------------------------
>
>                 Key: HDFS-3529
>                 URL: https://issues.apache.org/jira/browse/HDFS-3529
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, performance
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Trevor Robinson
>         Attachments: HDFS-3529.patch
>
>
> The write path currently makes several unnecessary data copies in order to go to and
from byte arrays. We can improve performance by using direct byte buffers to avoid the copy.
This is also a prerequisite for native checksum calculation (HDFS-3528)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message