hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2656) Implement a pure c client based on webhdfs
Date Tue, 11 Sep 2012 18:02:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453237#comment-13453237

Colin Patrick McCabe commented on HDFS-2656:

bq. For hdfsCopy/hdfsMove, any comments about their implementation without JNI?

In the JNI-based libhdfs, {{hdfsCopy}} calls {{FileUtil#copy}}.  This method basically just
does what you would expect-- it opens an input stream for the input file(s), and copies the
bytes to the output file(s).

Probably the most complicated part of {{FileUtil#copy}} is that it works recursively when
used on directories.  However, there's no special atomicity semantics when {{FileUtil#copy}}
copies a directory-- it just iterates through the files in a directory and copies what it
can.  If the directory is updated while it's iterating, that may or may not be reflected in
the copy.  {{hdfsMove}} basically just invokes {{FileUtil#copy}} with {{deleteSource}} = true.

So I think we should implement these methods in libwebhdfs using the methods which are already
> Implement a pure c client based on webhdfs
> ------------------------------------------
>                 Key: HDFS-2656
>                 URL: https://issues.apache.org/jira/browse/HDFS-2656
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>            Reporter: Zhanwei.Wang
>            Assignee: Jing Zhao
>             Fix For: 2.0.3-alpha
>         Attachments: HDFS-2656.patch, HDFS-2656.patch, HDFS-2656.patch, HDFS-2656.unfinished.patch,
> Currently, the implementation of libhdfs is based on JNI. The overhead of JVM seems a
little big, and libhdfs can also not be used in the environment without hdfs.
> It seems a good idea to implement a pure c client by wrapping webhdfs. It also can be
used to access different version of hdfs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message