hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3721) hsync support broke wire compatibility
Date Wed, 25 Jul 2012 03:24:35 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HDFS-3721:
------------------------------

    Attachment: hdfs-3721.txt

This patch fixes the issue as follows:
- Refactors out the packet-reading code from BlockReceiver and RemoteBlockReader2 into a new
{{PacketReceiver}} class. This really simplified BlockReceiver in particular, and has the
nice side effect of getting us significantly closer to HDFS-3529.
- All places where we used to assume a fixed-length packet header now support variable length.
In some cases this is achieved by allocating a larger-than-necessary buffer and then, once
we know the size for the header, putting it at the right spot to make the header contiguous
with the data. In other cases, this is achieved by simply separating the buffer containing
the header from the buffer containing the data.

- Regarding the issue above with 2.1 clients writing to 2.0 servers, I fixed the issue by
having the PacketHeader class not set any value for the {{syncBlock}} flag when it is false.
That means that, so long as the new hsync functionality isn't used, new clients can still
talk to 2.0 servers. If hsync is used, an error will occur. It's slightly unfortunate, but
given that the 2.0 branch is pretty new, and this is a new feature. I think this is acceptable.

I manually tested a trunk client both reading and writing from a 2.0.0-alpha cluster with
this patch applied.
                
> hsync support broke wire compatibility
> --------------------------------------
>
>                 Key: HDFS-3721
>                 URL: https://issues.apache.org/jira/browse/HDFS-3721
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node, hdfs client
>    Affects Versions: 2.1.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-3721.txt
>
>
> HDFS-744 added support for hsync to the data transfer wire protocol. However, it actually
broke wire compatibility: if the client has hsync support but the server does not, the client
cannot read or write data on the old cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message