hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
Date Thu, 01 Apr 2010 20:50:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852541#action_12852541
] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

> A design document as Jeff suggested would be useful.

What's unspecified in current issues?  If you like I can collate the comments on the various
issues linked here into a single document if that would make them more readable for you.

> I assume that the goal is still to make the RPC completely pluggable. i.e 'til the AVRO
path is stable, the default config uses the Hadoop RPC with the old serialization and that
there is no -ve performance impact on the
old code paths.

RPC is already pluggable w/o negative performance impact.  That was done in HADOOP-6422.

I don't expect that switching to Avro serialization for RPCs will affect performance, but
that should certainly be tested before we make Avro serialization the default.  Switching
the transport is more likely to affect performance, but that switch can be made separately
and after switching serializations.

So I see something like the following steps:
 - HDFS
 -- get hdfs tests to pass using Avro RPC serialization
 -- test hdfs performance using Avro RPC serialization
 -- switch HDFS to use Avro RPC serialization by default 
 -- design, implement and switch HDFS to use IDL-driven Avro RPC (HDFS-1069)
 - MapReduce
 -- get mapreduce tests to pass using Avro RPC serialization
 -- test mapreduce performance using Avro RPC serialization
 -- switch Mapreduce to use Avro RPC serialization by default
 -- design, implement and switch MapReduce to use IDL-driven Avro RPC (HDFS-1069)
 - Transport
  -- Design and develop an interoperable, secure, high-performance Avro transport (AVRO-341)
  -- port HDFS and MapReduce to use this optionally
  -- test HDFS and mapreduce with this new transport
  -- switch HDFS and Mapreduce to use new transport by default

Would it be useful to file a Jira issue for each of these?

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message