hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4136) imeplement DFSClient on top of thriftfs - this may require DFSClient or DN protocol changes
Date Tue, 09 Sep 2008 21:15:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629620#action_12629620

Pete Wyckoff commented on HADOOP-4136:

It's nice to be able to change the APIs under one without warning. I guess what I'm asking
for is something maybe for more like 1.0+ that opens the APIs so I can write applications
that take advantage of them (maybe we aren't there yet)

One big one could be a C++ dfsclient.  that's a lot of work though (and thus the idea of 
a Java proof of concept since we have the code already). I know there is talking about changing
the RPC and I guess this JIRA expresses the idea that those RPCs should be open and cross
language compatible. I often find myself in situations where I want something from Hadoop
but can't link the whole thing with my application. Admittedly this is mostly management type

If I had a pure C++ DFSClient, I could implement a FUSE (or direct kernel module) that talks
to HDFS without any Java at all.  For the kernel case, obviously this would be critical.

I think having a C++ DFSClient would open up a lot of things.

-- pete

> imeplement DFSClient on top of thriftfs - this may require DFSClient or DN protocol changes
> -------------------------------------------------------------------------------------------
>                 Key: HADOOP-4136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4136
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/thiftfs, dfs, fs
>            Reporter: Pete Wyckoff
>            Priority: Minor
> Open up DFS Protocol to allow non-Hadoop DFS clients to implement reads/writes.  Obviously,
the NN need not be changed because the thriftfs server will serve up the same metadata - ie
it's a bridge to the NN.
> This is useful because if we can do this in Java using more open APIs, we could do it
in C++ or Python or Perl :)
> Doing it in Java first makes sense because we already have the DFSClient - kind of a
proof of concept.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message