hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11671) Asynchronous native RPC v9 client
Date Wed, 11 Mar 2015 19:55:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357468#comment-14357468

Haohui Mai commented on HADOOP-11671:

bq. I don't want to be a kill-joy, but we now have 3 branches with native client RPC implementations,
and 0 usable native clients. I hope that we can integrate this efforts with the HDFS-6994
effort and deliver something that works.

Agree. Just to echo my previous comment, one of the motivation of this jira is to salvage
some parts from both HDFS-6994 and HADOOP-10388 branch while fixing all the issues that we
are uncomfortable with (see my previous comments). With all the previous investments on Hadoop
RPC I hope we can get this part merged in quickly.

> Asynchronous native RPC v9 client
> ---------------------------------
>                 Key: HADOOP-11671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11671
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>              Labels: native, rpc
> There are more and more integration happening between Hadoop and applications that are
implemented using languages other than Java.
> To access Hadoop, applications either have to go through JNI (e.g. libhdfs), or to reverse
engineer the Hadoop RPC protocol. (e.g. snakebite). Unfortunately, neither of them are satisfactory:
> * Integrating with JNI requires running a JVM inside the application. Some applications
(e.g., real-time processing, MPP database) does not want the footprints and GC behavior of
the JVM.
> * The Hadoop RPC protocol has a rich feature set including wire encryption, SASL, Kerberos
authentication. Many 3rd-party implementations can fully cover the feature sets thus they
might work in limited environment.
> This jira is to propose implementing an Hadoop RPC library in C++ that provides a common
ground to implement higher-level native client for HDFS, YARN, and MapReduce.

This message was sent by Atlassian JIRA

View raw message