hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
Date Wed, 24 Mar 2010 22:59:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849516#action_12849516

Doug Cutting commented on HADOOP-6659:

Avro RPC will enhance Hadoop in two primary areas:
 - versioning: Hadoop's current RPC mechanism requires that clients and servers evolve in
lockstep.  Avro RPC permits clients and servers using different versions of protocols to still
communicate.  Avro doesn't automatically solve all versioning problems, but it does give one
considerably better tools to solve them.
 - language-independence: Hadoop's RPC is currently Java-only, while Avro supports Java, Python,
Ruby, C and C++ so far.  A long-term goal for Hadoop is to permit native implementations of
certain RPC protocols in other languages, e.g., perhaps mapreduce job submission and hdfs

The intended approach is to initially use Avro's reflection API tunneled over Hadoop RPC:
 - Reflection is used initially to minimize the impact on the codebase.  Subsequently we should
consider, protocol-by-protocol, switching to IDL-driven protocols, generating Java APIs using
Avro specific.
- Tunneling is used initially to leverage Hadoop's existing high-performance, secure RPC transport.
 Once Avro has an equivalent (AVRO-341) we can consider switching Hadop to use to that, to
achieve language-independence.

Once reflection works correctly then we will have an Avro-based RPC protocol specification
that correclty models Hadoop's protocols.  This alone would address Hadoop's versioning issues,
but it would not address the language-interoperability, since the Hadoop RPC tunnel used is
not a cross-language standard.  For language interoperability, we need AVRO-341.

With HADOOP-6422 and HDFS-892, HDFS tests can now be run using Avro reflection.  Some basic
tests pass but many still fail.  These failures will be addressed in HDFS-1066.

> Switch RPC to use Avro
> ----------------------
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message