hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7887) Asynchronous native RPC v9 client
Date Thu, 05 Mar 2015 02:03:38 GMT
Haohui Mai created HDFS-7887:

             Summary: Asynchronous native RPC v9 client
                 Key: HDFS-7887
                 URL: https://issues.apache.org/jira/browse/HDFS-7887
             Project: Hadoop HDFS
          Issue Type: New Feature
            Reporter: Haohui Mai
            Assignee: Haohui Mai

There are more and more integration happening between Hadoop and applications that are implemented
using languages other than Java.

To access Hadoop, applications either have to go through JNI (e.g. libhdfs), or to reverse
engineer the Hadoop RPC protocol. (e.g. snakebite). Unfortunately, neither of them are satisfactory:

* Integrating with JNI requires running a JVM inside the application. Some applications (e.g.,
real-time processing, MPP database) does not want the footprints and GC behavior of the JVM.
* The Hadoop RPC protocol has a rich feature set including wire encryption, SASL, Kerberos
authentication. Many 3rd-party implementations can fully cover the feature sets thus they
might work in limited environment.

This jira is to propose implementing an Hadoop RPC library in C++ that provides a common ground
to implement higher-level native client for HDFS, YARN, and MapReduce.

This message was sent by Atlassian JIRA

View raw message