hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2060) DFS client RPCs using protobufs
Date Fri, 10 Jun 2011 17:48:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047324#comment-13047324
] 

Todd Lipcon commented on HDFS-2060:
-----------------------------------

Here's my proposal:
- in a HADOOP jira, modify ObjectWritable to be able to serialize protocol buffers (ie the
{{Message}}) type. This allows coexistence of writables and protobufs within a single protocol
so we can do a staged switchover without a wholesale switch in rpc engine implementation.
- for each of the RPCs in ClientProtocol:
-- define a new protobuf for the request params and the resposne params. EG {{SetReplicationRequestProto}}
and {{SetReplicationResponseProto}}.
-- add a new RPC {{SetReplicationResponseProto setReplication(SetReplicationResponseProto
param)}}
-- implement that RPC by simply writing the boilerplate wrap/unwrap code to/from the original
parameters
-- deprecate the old call
-- switch callers internal to HDFS to the new proto-based call

The nice thing about this approach is that it (a) maintains API backwards-compatibility for
one version for folks directly using protocol proxies to talk to the NN, and (b) allows us
to do a staged switchover, one RPC at a time.

It doesn't address the ability to evolve the underlying IPC/RPC transport, but that could
be tackled in parallel or sequenced after this.

> DFS client RPCs using protobufs
> -------------------------------
>
>                 Key: HDFS-2060
>                 URL: https://issues.apache.org/jira/browse/HDFS-2060
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>
> The most important place for wire-compatibility in DFS is between clients and the cluster,
since lockstep upgrade is very difficult and a single client may want to talk to multiple
server versions. So, I'd like to focus this JIRA on making the RPCs between the DFS client
and the NN/DNs wire-compatible using protocol buffer based serialization.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message