hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
Date Mon, 05 Jan 2015 17:34:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264800#comment-14264800
] 

Colin Patrick McCabe commented on HDFS-6994:
--------------------------------------------

I don't think you need to do so much work.  Just use the existing native {{MiniDFSCluster}}
code.  Then, you can provide a mode where you can pass in the URL of a real HDFS cluster to
the test and have it use that instead of starting its own JVM.  This allows you to avoid JNI,
but also run the test in a self-contained fashion.

The amount of interaction we do with {{MiniDFSCluster}} is pretty minimal and the JNI code
is already written.  If we start writing new inter-process communication protocols, we will
have to debug those as well.  Since JSON is weakly typed, we would have all the type coercion
sloppiness issues of JNI and then some.  Managing external processes is also annoying, especially
on Windows (which we are trying to support).

> libhdfs3 - A native C/C++ HDFS client
> -------------------------------------
>
>                 Key: HDFS-6994
>                 URL: https://issues.apache.org/jira/browse/HDFS-6994
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client
>            Reporter: Zhanwei Wang
>            Assignee: Zhanwei Wang
>         Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch
>
>
> Hi All
> I just got the permission to open source libhdfs3, which is a native C/C++ HDFS client
based on Hadoop RPC protocol and HDFS Data Transfer Protocol.
> libhdfs3 provide the libhdfs style C interface and a C++ interface. Support both HADOOP
RPC version 8 and 9. Support Namenode HA and Kerberos authentication.
> libhdfs3 is currently used by HAWQ of Pivotal
> I'd like to integrate libhdfs3 into HDFS source code to benefit others.
> You can find libhdfs3 code from github
> https://github.com/PivotalRD/libhdfs3
> http://pivotalrd.github.io/libhdfs3/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message