hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-10389) Native RPCv9 client
Date Thu, 27 Mar 2014 02:31:17 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Colin Patrick McCabe updated HADOOP-10389:

    Attachment: HADOOP-10388.001.patch

This patch implements an async Hadoop RPC client in C.  It uses libuv and protobuf-c.

CMakeLists.txt: the top-level build file which specifies build products.

GenerateProtobufs.cmake: this CMake script invokes the protobuf compiler on the various Hadoop
protobuf files.

common/hadoop_err.c: implements hadoop error objects which can be easily created and managed
to represent errors throughout the system.  They have both an error code and an error message.
 The error message starts with an exception name.

common/net.c: network utilities.  This is very minimal since libuv provides most of what we

common/test.h: unit test macros

common/tree.h: BSD-licensed red-black tree, similar to what we use in libhdfs.

common/queue.h: BSD-licensed list implementation

hdfs/namenode-rpc-unit.c: end-to-end test of the RPC framework.

rpc/call.c: represents an in-progress RPC call.

rpc/client_id.c: represents a 16-byte client ID, as specified in Hadoop RPC.

rpc/conn.c: represents an existing TCP connection to a remote Hadoop server.

rpc/messenger.c: the messenger coordinates all network traffic.  It owns all the reactor threads.
 Currently there is only one reactor thread, but in the future we may want more than one to
get more parallelism.

rpc/proxy.c: clients use the proxy to start and manage RPCs, similar to how Proxy is used
in the Hadoop java code.  There is going to be some more glue code here so that each Hadoop
RPC will have an equivalent sync and async invocation function.

rpc/reactor.c: a reactor is a single thread calling epoll.  The messenger owns all reactors.
 Connection callbacks happen in the context of the reactor thread.

rpc/shorten.c: shorten is a utility which makes some really long object names generated by
protobuf-c shorter.

rpc/varint.c: varint handles the serialization and deserialization of variable length integers.

Build with "cmake . && make"
Unit tests can be run with "make test."

> Native RPCv9 client
> -------------------
>                 Key: HADOOP-10389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10389
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: HADOOP-10388
>            Reporter: Binglin Chang
>            Assignee: Colin Patrick McCabe
>         Attachments: HADOOP-10388.001.patch

This message was sent by Atlassian JIRA

View raw message