hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7557) Make IPC header be extensible
Date Wed, 12 Oct 2011 20:39:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126128#comment-13126128

Doug Cutting commented on HADOOP-7557:

Todd, I mostly agree with you.

There are a few goals one might have for RPC in Hadoop:
  # support compatibility across versions of Hadoop, permitting rolling upgrades, etc.
  # permit RPC clients and servers in languages besides Java
  # change RPC, Shuffle and HDFS to use common communications layer, simplifying the implementation
and evolution of security, performance, etc.

To my understanding Sanjay is only currently pursuing the first, and in particular wants to
be able to provide compatibility between 0.20, 0.23 and 0.24  To do this, he'd like to permit
both an old serialization (Writable) and a new serialization (Protobuf).

I am interested in all three, and would propose that we not attempt to implement wire-compatibility
with 0.20 and perhaps not even with 0.23, but rather select a single message serialization
and client/server protocol that's programming language-independent and only promise wire-compatibility
between releases once we've made a transition to those.  I don't believe that anything is
language independent until it has implementations in multiple languages.  The more we add
features to Hadoop's existing RPC framework the further we get from it supporting multiple
languages and using a standard client/server protocol.

My interest in this particular issue is simply to try to keep the communications layer something
that we might someday reasonably describe as a network standard, i.e., comparable to SASL
or HTTP.
> Make  IPC  header be extensible
> -------------------------------
>                 Key: HADOOP-7557
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7557
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: HADOOP-7557.patch, IpcHeader.proto, ipcHeader1.patch, ipcHeader2.patch

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message