Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 5 Sep 2014 21:37:29 +0000 (UTC)
From: "Colin Patrick McCabe (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12739026.1409812767000.5874.1409953049890@Atlassian.JIRA>
In-Reply-To: <JIRA.12739026.1409812767000@Atlassian.JIRA>
References: <JIRA.12739026.1409812767000@Atlassian.JIRA>
 <JIRA.12739026.1409812767202@arcas>
Subject: [jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS
 client
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123627#comment-14123627 ] 

Colin Patrick McCabe commented on HDFS-6994:
--------------------------------------------

Hi Zhanwei, this is really interesting.  As Wenwu mentioned, there's already a branch where we're working on a native client.  It would be nice if we could integrate this with that work somehow.  I'm not sure what form that should take.

Did you get a chance to read the design doc on HADOOP-10388?  There are a few issues important here before this can replace libhdfs.  We need the ability to fall back to the JNI code when necessary-- for example, in the case where HDFS is using encryption, and we don't have native client support for that.  But we don't want a hard library dependency on libjvm.so-- it should be dynamically loaded.

It's good that you are using the existing hdfs.h interface.  getLastError seems like it could be a good addition, as well... when using thread-local data for the string.

The dependencies here are problematic.  libxml2 is not fully thread-safe, and it pulls in a lot of GNOME stuff we don't really want.  The boost dependency creates problems as well.  For example, Impala depends on a certain version of boost-- if this library pulls in a different version, bad things happen.  GnuTls is LGPL, which makes it difficult to ship.  I would have to be -1 just based on the dependencies alone...

We also have duplicated protobuf files in this patch.  We should simply use the protobuf files in the source tree already.

If I could summarize my first thoughts:
* get rid of boost, including all boost ifdefs
* don't worry about earlier RPC versions... we only need to support RPCv9 now (same as Java client code policy in Hadoop)
* use libexpat or something instead of libxml2

This is good work overall and hopefully there is stuff we can use here.

> libhdfs3 - A native C/C++ HDFS client
> -------------------------------------
>
>                 Key: HDFS-6994
>                 URL: https://issues.apache.org/jira/browse/HDFS-6994
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: hdfs-client
>            Reporter: Zhanwei Wang
>         Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch
>
>
> Hi All
> I just got the permission to open source libhdfs3, which is a native C/C++ HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.
> libhdfs3 provide the libhdfs style C interface and a C++ interface. Support both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos authentication.
> libhdfs3 is currently used by HAWQ of Pivotal
> I'd like to integrate libhdfs3 into HDFS source code to benefit others.
> You can find libhdfs3 code from github
> https://github.com/PivotalRD/libhdfs3


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)