hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Binglin Chang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
Date Tue, 30 Dec 2014 07:13:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260874#comment-14260874

Binglin Chang commented on HDFS-6994:

About adding more tests, we should add minidfscluster support, we can reuse native_mini_dfs.h
in libhdfs, but it has some limitations:
1. it lacks some functionalities to do all the tests. e.g. start/stop datanode, corrupt file.
2. it add dependency of jni
3. add method support in native minidfscluster involve lot of work(get method id, type conversion
I have another idea of doing this:
1. add some cli like interface to MiniDFSCluster in java side. support most commonly used
MiniDFSCluster method as cli commands should be easy using reflection and json
2. On libhdfs3 side, tests can start MiniDFSCluster cli process and call those method in a
cli+json protocol
If you guys thinks its OK, I can create a task and work on this.

> libhdfs3 - A native C/C++ HDFS client
> -------------------------------------
>                 Key: HDFS-6994
>                 URL: https://issues.apache.org/jira/browse/HDFS-6994
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client
>            Reporter: Zhanwei Wang
>            Assignee: Zhanwei Wang
>         Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch
> Hi All
> I just got the permission to open source libhdfs3, which is a native C/C++ HDFS client
based on Hadoop RPC protocol and HDFS Data Transfer Protocol.
> libhdfs3 provide the libhdfs style C interface and a C++ interface. Support both HADOOP
RPC version 8 and 9. Support Namenode HA and Kerberos authentication.
> libhdfs3 is currently used by HAWQ of Pivotal
> I'd like to integrate libhdfs3 into HDFS source code to benefit others.
> You can find libhdfs3 code from github
> https://github.com/PivotalRD/libhdfs3
> http://pivotalrd.github.io/libhdfs3/

This message was sent by Atlassian JIRA

View raw message