hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2656) Implement a pure c client based on webhdfs
Date Thu, 09 Aug 2012 17:02:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431974#comment-13431974

Suresh Srinivas commented on HDFS-2656:

Some comments:
bq. I am uncomfortable calling this "libhdfs2" if the performance is not up to par with the
current "libhdfs1". 
I understand calling it libwebhdfs for better clarity. But the above reason, based on performance
does not make much sense to me.

bq. @Eli What's your use case?
I am very surprised by this. You made a case of this in the related jira HDFS-2631, [here|https://issues.apache.org/jira/browse/HDFS-2631?focusedCommentId=13255263&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13255263]
and [here|https://issues.apache.org/jira/browse/HDFS-2631?focusedCommentId=13408375&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13408375].
I know of several people building there own libraries for webhdfs interfaces. This would provide
one in Hadoop that they can use. This will also help us understand the performance differences
between JNI based implementation vs webhdfs.

I agree libhdfs3 that directly uses protobuf interfaces is the most useful. But DFSClient
today is a very think client with all kinds of sophisticated logic put into it, that uses
both RPC and DataTransferProtocol. As we speak it keeps becoming more sophisticated (see HDFS-3672).
I am not sure of seeing a complete, robust implementation in C in the near term. I also do
not think c implementation can keep up with the changes we keep making in DFSClient.

> Implement a pure c client based on webhdfs
> ------------------------------------------
>                 Key: HDFS-2656
>                 URL: https://issues.apache.org/jira/browse/HDFS-2656
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>            Reporter: Zhanwei.Wang
>         Attachments: HDFS-2656.patch, HDFS-2656.unfinished.patch
> Currently, the implementation of libhdfs is based on JNI. The overhead of JVM seems a
little big, and libhdfs can also not be used in the environment without hdfs.
> It seems a good idea to implement a pure c client by wrapping webhdfs. It also can be
used to access different version of hdfs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message