hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7557) Make IPC header be extensible
Date Fri, 10 Feb 2012 19:21:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205663#comment-13205663

Doug Cutting commented on HADOOP-7557:

Sanjay> [RpcEngine] can help us provide 20 compatibility

I'm confused.  If you switch the RPC headers to protobuf how can you provide 20 compatibility?
 If the protobuf headers are confined to ProtobufRpcEngine then I have no issues.  I thought
the proposal here is to change RPC format so that all RpcEngine implementations use protobuf
headers.  That would break 0.20 compatibility, no?

Sanjay> Another project can use Hadoop RPC but a different serialization then it can. (Say
avro or thrift).

If we don't have multiple implementations of an extension API that are regularly used then
the chances that that it's actually possible to create other implementations that work is
small.  The reason we removed Avro was that implementing RpcEngine alone was no longer sufficient:
there were assumptions that Protobuf was used in other places in the code.  RpcEngine is currently
a broken abstraction, providing a false claim of plugability.

Sanjay> Doug can add his k-v pair as a parallel structure if he wishes.

I have no desire to add a k-v parallel structure.  I'm just trying to help keep the RPC architecture
simple and sane.

Suresh> it seems to me that all are optional except EOH

The protobuf documentation says, "Some engineers at Google have come to the conclusion that
using required does more harm than good; they prefer to use only optional and repeated. However,
this view is not universal."  So (arguably) all or most fields should be optional.

Suresh> Documenting header - Hadoop documentation has been poor and incomplete.

Indeed.  If Hadoop RPC is to be interoperable then it needs a language-independent specification
and multiple implementations.  That specification might specify its headers either as some
protobuf messages or as some RFC-like text.  Either would work fine.  The default value for
optional header fields could be specified in either.  But what's also required is a description
of how to handle the various values for these header fields, including the default.  Until
then, the implementation is the specification.

> Make  IPC  header be extensible
> -------------------------------
>                 Key: HADOOP-7557
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7557
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: HADOOP-7557.patch, IpcHeader.proto, ipcHeader1.patch, ipcHeader2.patch

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message