hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
Date Fri, 10 Jun 2011 20:01:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047406#comment-13047406

Arun C Murthy commented on HDFS-2058:

Protobuf has two nice features that thrift doesn't have yet: 1) when unknown data is read,
it is maintained in a map and then put back on the wire if the same object is rewritten. 2)
it has a decent plugin system that makes it easy to modify the generated code – even with
a plugin written in python or Java, in theory. These could be implemented in Thrift, but again,
I didn't want to take the time.

Similar reasons to choose PB for MRv2. Also, MRv2 has a pluggable layer, so in theory, it
could be anything. We choose PB for exact same reasons outlined by Todd.

But, I'd encourage keeping serialization and in-memory data-structures separate ala MR-279.

Mainly, we need to move ahead with something.


> DataTransfer Protocol using protobufs
> -------------------------------------
>                 Key: HDFS-2058
>                 URL: https://issues.apache.org/jira/browse/HDFS-2058
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>         Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt
> We've been talking about this for a long time... would be nice to use something like
protobufs or Thrift for some of our wire protocols.
> I knocked together a prototype of DataTransferProtocol on top of proto bufs that seems
to work.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message