Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Thu, 27 Mar 2014 02:31:17 +0000 (UTC)
From: "Colin Patrick McCabe (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12699067.1394079295281.13803.1395887477249@arcas>
In-Reply-To: <JIRA.12699067.1394079295281@arcas>
References: <JIRA.12699067.1394079295281@arcas>
Subject: [jira] [Updated] (HADOOP-10389) Native RPCv9 client
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HADOOP-10389:
------------------------------------------

    Attachment: HADOOP-10388.001.patch

This patch implements an async Hadoop RPC client in C.  It uses libuv and protobuf-c.

CMakeLists.txt: the top-level build file which specifies build products.

GenerateProtobufs.cmake: this CMake script invokes the protobuf compiler on the various Hadoop protobuf files.

common/hadoop_err.c: implements hadoop error objects which can be easily created and managed to represent errors throughout the system.  They have both an error code and an error message.  The error message starts with an exception name.

common/net.c: network utilities.  This is very minimal since libuv provides most of what we need.

common/test.h: unit test macros

common/tree.h: BSD-licensed red-black tree, similar to what we use in libhdfs.

common/queue.h: BSD-licensed list implementation

hdfs/namenode-rpc-unit.c: end-to-end test of the RPC framework.

rpc/call.c: represents an in-progress RPC call.

rpc/client_id.c: represents a 16-byte client ID, as specified in Hadoop RPC.

rpc/conn.c: represents an existing TCP connection to a remote Hadoop server.

rpc/messenger.c: the messenger coordinates all network traffic.  It owns all the reactor threads.  Currently there is only one reactor thread, but in the future we may want more than one to get more parallelism.

rpc/proxy.c: clients use the proxy to start and manage RPCs, similar to how Proxy is used in the Hadoop java code.  There is going to be some more glue code here so that each Hadoop RPC will have an equivalent sync and async invocation function.

rpc/reactor.c: a reactor is a single thread calling epoll.  The messenger owns all reactors.  Connection callbacks happen in the context of the reactor thread.

rpc/shorten.c: shorten is a utility which makes some really long object names generated by protobuf-c shorter.

rpc/varint.c: varint handles the serialization and deserialization of variable length integers.

Build with "cmake . && make"
Unit tests can be run with "make test."

> Native RPCv9 client
> -------------------
>
>                 Key: HADOOP-10389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10389
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: HADOOP-10388
>            Reporter: Binglin Chang
>            Assignee: Colin Patrick McCabe
>         Attachments: HADOOP-10388.001.patch
>
>


--
This message was sent by Atlassian JIRA
(v6.2#6252)