hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
Date Fri, 09 Apr 2010 16:28:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855460#action_12855460
] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

Eric, this work is proceeding using a "branch in place" strategy, since long-lived branches
are onerous to mange.  The intent is to be largely equivalent to a branch, in that existing
code will not be disturbed.  The first step was to make the RPC functionality we wish to replace
pluggable.  That was committed last September in HADOOP-6170.  Now we can develop and test
the new RPC implementation without disturbing the existing implementation.  There is no intent
to switch to the new implementation until all concerns about it are resolved.  Some minor,
non-functional changes have and will be made to the existing implementation to aid IDL inference,
i.e. adding some annotations and renaming some private methods in (HDFS-1077).

> Until your proposal has been formally documented, including an IDL, which I think is
critical to the stated goal of backward compatibility, I'm not going to be comfortable seeing
it committed.

This issue does not attempt to fully implement cross-version wire-compatiblity.  Nor does
it attempt to move Hadoop to IDL-driven RPC.  Rather it proposes to move Hadoop to a system
that we can use to implement wire-compatibility and IDL-driven RPC.

Until we switch, we cannot dictate the IDL, the IDL is derived from the existing Java interfaces
used to define RPC protocols.  Once we've switched to the new Avro-based implementation, then
we can elect to, protocol-by-protocol, client-by-client and server-by-server, switch to an
IDL-driven implementation.  (An IDL derived by the current Java interfaces is attached to
HDFS-1069.  This is in JSON form, but Avro has an alternate IDL syntax that, by the time we
consider switching, would be used instead.)

Changes to the protocol often imply changes to client and server logic.  This issue does not
propose to alter client and server logic, but rather to track and retain the existing logic
and protocols.

Before we make a release that we claim will support wire-compatibility we should certainly
review our protocols carefully.  But first we need to switch an RPC system that permits clients
and servers to use different protocol versions and that supports IDL-driven protocols.  That's
the primary focus of this issue.  Once we switch, our wire-compatibility problems will not
be automatically solved, rather we'll then have tools in place to address them.

> Beyond understanding the design, you need a testing plan and test results to review.

Yes, I agree, this should be fully tested to everyone's satisfaction before we make any switch.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message