hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "LibHDFS" by Arun C Murthy
Date Fri, 25 Aug 2006 05:03:30 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by Arun C Murthy:

The comment on the change is:
First cut

New page:
= libhdfs =

 ''''libhdfs'''' is a [http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/jniTOC.html JNI]
based C api for Hadoop's DFS. 
 It provides a simple subset of C apis to manipulate DFS files and the filesystem. libhdfs
is available for [http://www.apache.org/dyn/closer.cgi/lucene/hadoop/ download] as a part
of Hadoop itself. The source for libhdfs is available for browsing [http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/c%2B%2B/libhdfs/

==== Table Of Contents ====
 1. [#Overview Overview]
 2. [#Setup Setup]
 3. [#Apis APIs]
  a. [#FileSystemApis FileSystem Manipulation]
  b. [#FileApis File Manipulation]
  c. [#Threading Support for Threads]
 4. [#Contact Assistance/Bugs]
 5. [#Conclusion Conclusion]

===== Overview =====
 [http://wiki.apache.org/lucene-hadoop/DFS Here] is an overview of Hadoop's DFS. The javadocs
for DFS are available [http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/fs/package-summary.html
here] and [http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/dfs/package-summary.html
here]. '''libdhfs''' is a simple JNI based C api for accessing and manipulating Hadoop's DFS
from native code. It offers a simple subset of the same functionality.

===== Setup =====
 It is necessary to setup Hadoop's DFS itself first. The information to setup Hadoop is available
[http://lucene.apache.org/hadoop/docs/api/overview-summary.html#overview_description here].
Once you have a working setup, you will need to get into the '''src/c++/libhdfs''' directory
and use the Makefile to build libhdfs (in case of issues use [#Contact this]). Once you have
successfully built libhdfs you can link it into your programs and are good to go.

===== APIs =====
 This section describes the various apis provided by '''libhdfs''' to manipulate the DFS.
It is classified into apis which manipulate individual files and those which manipulate the
filesystem itself. (Please see the doxygen documentation [# here] for details of individual

''!FileSystem APIs''
 libhdfs provides apis for both '''generic''' manipulation of the filesytem (create directories,
copy/move files etc.) and also some very DFS specific functionality (get information on file
replication etc.).
 At startup one should use the ''hdfsConnect'' api to connect to the DFS before any operations
can be performed (on files or the filesystem); the analogous ''hdfsDisconnect'' performs a
clean teardown of the connection.
 Generic operations:
 * hdfsCopy (across filesytems also)
 * hdfsMove (across filesytems also)
 * hdfsRename
 * hdfsDelete
 libhdfs also provides apis for manipulating directories on the DFS:
 * hdfsCreateDirectory
 * hdfsSetWorkingDirectory
 * hdfsGetWorkingDirectory
 * hdfsListDirectory / hdfsGetPathInfo / hdfsFreeFileInfo
 The apis to query the filesytems for various properties:
 * hdfsGetHosts
 * hdfsGetDefaultBlockSize
 * hdfsGetUsed / hdfsGetCapacity
''File APIs''
 libhdfs provides posix-like apis to manipulate individual files (create, read/write, query
etc.) listed below:
 * hdfsOpenFile / hdfsCloseFile
 * hdfsRead / hdfsWrite
 * hdfsTell / hdfsSeek
 * hdfsFlush
 * hdfsAvailable
 * hdfsLock / hdfsUnlock

''Using libhdfs in Threaded Applications''
 libhdfs can be used in threaded applications using the !Posix !Threads. However to carefully
interact with JNI's global/local references the user has to explicitly call the ''hdfsConvertToGlobalRef''
/ ''hdfsDeleteGlobalRef'' apis.

===== Contact Information =====

 Please drop us an email at '''hadoop-users@lucene.apache.org''' if you have any questions
or any suggestions. Use [http://issues.apache.org/jira/browse/HADOOP Jira]] to report bugs.

===== Conclusion =====

 Thank you for your interest in Hadoop and libhdfs!

View raw message