hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2656) Implement a pure c client based on webhdfs
Date Tue, 11 Sep 2012 01:37:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452638#comment-13452638
] 

Colin Patrick McCabe commented on HDFS-2656:
--------------------------------------------

I'm glad we managed to get this merged.  Good job, Jing!

I think there's some refactoring we can do to improve it further.  Please take a look at HDFS-3916
when you get a chance.

Also re: JNI-- I think we might be better off without it.  One of the big benefits of webhdfs
is that you can update the server components without changing the software on the clients.
 Because it's a stable REST API, everything will "just work"-- no need to roll out new JAR
files to every client.  That benefit is lost if we start using JNI in libwebhdfs.

We might be able to read the XML preference files with an XML parsing library, and determine
the default NameNode and port this way.  After all, Hadoop preference files are just XML files.
                
> Implement a pure c client based on webhdfs
> ------------------------------------------
>
>                 Key: HDFS-2656
>                 URL: https://issues.apache.org/jira/browse/HDFS-2656
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>            Reporter: Zhanwei.Wang
>            Assignee: Jing Zhao
>             Fix For: 2.0.3-alpha
>
>         Attachments: HDFS-2656.patch, HDFS-2656.patch, HDFS-2656.patch, HDFS-2656.unfinished.patch,
teragen_terasort_teravalidate_performance.png
>
>
> Currently, the implementation of libhdfs is based on JNI. The overhead of JVM seems a
little big, and libhdfs can also not be used in the environment without hdfs.
> It seems a good idea to implement a pure c client by wrapping webhdfs. It also can be
used to access different version of hdfs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message