hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
Date Mon, 12 Apr 2010 18:55:56 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856123#action_12856123

Doug Cutting commented on HADOOP-6659:

> AVRO IDL-driven protocols are necessary for wire compatibility.

I agree.  As I said above, before we declare that we support wire-compatibility we should
perform a careful audit of our RPC protocols.  With the approach I've suggested, this can
largely be pursued in parallel.  Until we switch we can examine the reflected protocol and
work to improve (or at least develop a list of proposed improvements) the Java interfaces.

> what are benefits of the reflection scheme

By not branching we can continue development of HDFS and mapreduce in parallel while we develop
and test a new RPC serialization and transport.  As mentioned above, once we've switched to
Avro-serialization using reflect, then we can begin, protocol-by-protocol, switching each
to an IDL-based approach.  Each protocol can be addressed in a separate issue with no massive
branch required.  We might, e.g., prioritize client-facing protocols first, so that we can
support wire compatibility of clients before we support rolling cluster upgrades.  We can
even separate updating clients from updating servers.  Once we've completed the transition
to an IDL-driven system, then we can, protocol-by-protocol, method-by-method, work to improve
the IDL to the point where we're willing to declare our support of wire-compatibility.  At
no point is trunk broken or are large areas blocked from changes and fixes.

> Switch RPC to use Avro
> ----------------------
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message