hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
Date Thu, 29 May 2014 20:31:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012839#comment-14012839

Colin Patrick McCabe commented on HADOOP-10640:

With this patch, libhdfs.so continues to implement the same API as before.  At runtime, we
select which native filesystem implementation to use based on by looking at {{<scheme>.native.handler}}.
 So, if the scheme of the URI we're handling is HDFS, we'd look at {{hdfs.native.handler}}.
 If there is no scheme-specific handler, we fall back to {{default.native.handler}}.

There are two libhdfs native filesystems available at the moment.  One is jniFS, the existing
JNI-based filesystem.  The other is NDFS, the new native filesystem that we're developing.
 As mentioned earlier, jniFS has the advantage of working with things like s3 or HDFS encryption,
whereas NDFS has the advantage that it doesn't require JNI, is more efficient, etc.  The goal
here is to have a {{libhdfs.so}} that we can drop into existing installations and have everything
"just work."

The libhdfs.so library doesn't depend on JNI... it can be installed in cases where JNI is
not present.  The reason is because it uses {{dlopen}} (by means of the libuv shims) to open
the {{libjvm.so}} library as-needed.  So clients who don't want to install or use JNI don't
have to.

The build changed a little bit to have a {{config.h.cmake}} file which gets transformed into
{{config.h}} based on which features are present at compile-time.  This is how the existing
libhdfs build works (we need it to detect some platform-specific features.)  I also hide all
library symbols except for the public symbols, so that our library internals are not visible
to the outside world, using {{fvisibility=hidden}}.

source overview:
* common/hadoop_err: added the {{hadoop_err_prepend}}, {{hadoop_err_copy}}, and {{terror}}
utility functions.
* common/hconf: this is a native implementation of the Java Configuration class which reads
XML files.  It uses libexpat.
* common/htable: a hash table used in a few places.
* common/net: added {{net_ipv4_name_and_port}}, {{get_first_ipv4_addr}} utility functions
* common/string: added the {{strdupto}} utility function plus a unit test.
* common/uri: parses HDFS-style URIs into scheme, authority, port, etc. using the uriparser
* fs/fs.c: the entry points for all libhdfs functions.  Contains a vtable used to call the
appropriate implementation function.
* fs/hdfs.h: the public API of libhdfs.  This is just a rename.
* jni/: all the stuff in this directory was moved from the previous libhdfs location.
* net/proxy, net/reactor: fixed a bug in the synchronous RPC code
* test/: I created a test directory paralleling the source directory.  Moved test.h to this

Running {{NAMENODE_URI=hdfs://localhost:6000 ./test_libhdfs_meta_ops}} will give an idea of
how this operates.

> Implement Namenode RPCs in HDFS native client
> ---------------------------------------------
>                 Key: HADOOP-10640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10640
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: native
>    Affects Versions: HADOOP-10388
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
> Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as
mkdir, rename, etc.

This message was sent by Atlassian JIRA

View raw message