hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
Date Thu, 22 Apr 2010 20:20:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859992#action_12859992

Doug Cutting commented on HADOOP-6659:

Sanjay> Moving to Avro IDL will change many data types in our servers and cannot be pluggable.
Sanjay> I don't think we can split the HDFS protocols as they use many common data types
(block-id, block-token, etc). 

I agree that it may not be easy to do this incrementally, but I think it would be much easier
than trying to do it in a branch.  For example, existing datatypes like Block that are used
in many protocols could be made to implement Avro's SpecificRecord interface so that both
IDL and reflect-based code can use them.  Then we could more easily consider porting protocols
one-by-one.  Once we've ported all protocols that use Block, then we could, as a single patch,
replace it with an IDL-generated version.

Eric> If we can get to agreement that the current RPC remains the default and that IDL
work happens in a branch and that only after we have a reasonably complete backwards compat
solution do we change to AVRO RPC by default, then I think we are good.

If Avro reflect-based RPC passes all tests, including performance, reliability, etc., why
wouldn't we switch to it?  Even without using an IDL, this would let client and server versions
differ.  We certainly don't want to encourage independent 3rd party implementations of protocols
until we're happy that the protocols are what we intend to support long-term, but I don't
yet see a reason not to switch once Avro-based RPC is functionally equivalent.

Eric> My concern is not with the already committed plugability, but with the change of
the default RPC and incremental change over of the protocols. Once that work starts, we are
committed to finishing it and we can not do that in the timeframe of the next release IMO.

I don't follow.  If we proceed incrementally, thoroughly testing changes before committing
them, then we should be able to release at any time, no?

Eric> Would we keep the plugable API once we make the transition?

I see no reason to keep it.

Folks are welcome to start IDL-based branch(es) if they prefer to operate that way.  In that
case, I will cease work on support for an incremental approach, as it would be redundant.
 I fear that, since such branches would be long-lived that they'll prove very painful to maintain,
especially if they make substantial changes to protocols.  We should not prohibit trunk changes
to the HDFS and MapReduce protocols.

> Switch RPC to use Avro
> ----------------------
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message