hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2181) Seperate HDFS wire protocol data types
Date Tue, 04 Oct 2011 05:31:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119907#comment-13119907

Sanjay Radia commented on HDFS-2181:

Yes Todd, you got it right. However I was struck a little bit by your order and that you didn't
mention the key item A below.
Hence I re-order your items and add (A). Also see my next comment which explains how I plan
to proceed. 
* A. Separate all the datatypes used for wire protocol (called R23 at this time) into package
protocolR23Compatible. These data types are not exposed to DFSClient or to the NN. DFSClient
and NN use their native HDFS types (which currently reside in o.a.h.hdfs.protocol.) This is
done as follows:
** [Your item 3:] For each of the current types (eg LocatedBlocks, etc), copy them into a
protocolR23Compatible package. For each of these classes, add a static method to convert to/from
the generic type in the .protocol package
* B. Create 2 translator classes: ClientXXProtocolTranslator and ClientXXProtocolServerSideTranslator.
These translate from to and from wire types R23 to the builtin types of HDFS. (ie mostly what
you label as item 4)
* C. NN and DFSCLient are not exposed to proxies or to R23 data types: 
**  [Your item 1] Move the construction of RPC proxies and wire objects into a "client side
translator" class.
**  [Your item 2] Pass a new ServerSideTranslator implementation to the RPC.getServer calls.
This ServerSideTranslator translates from wire types back to the generic protocol types.

If we want to add new protocol serialization (such as PB or Avro) we would add a new package
similar to protocolR23Compatible. I have added a placeholder for PB. We can also support multiple
protocols very easily and pick the right one based on SS (one critical piece is missing for
this - I will file the jira).

> Seperate HDFS wire protocol data types
> --------------------------------------
>                 Key: HDFS-2181
>                 URL: https://issues.apache.org/jira/browse/HDFS-2181
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: separateDataType1.patch, separateDataType10.patch, separateDataType11.patch,
separateDataType12.patch, separateDataType13.patch, separateDataType14.patch, separateDataType2.patch,
separateDataType6.patch, separateDataType7.patch, separateDataType8.patch, separateDataType9.patch

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message