hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4136) imeplement DFSClient on top of thriftfs - this may require DFSClient or DN protocol changes
Date Tue, 09 Sep 2008 20:37:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629598#action_12629598

Pete Wyckoff commented on HADOOP-4136:

an approach could be:

1. implement the FileSystem API on top of thriftfs (even if we have to pull in all of Hadoop,
doesn't matter as long as we're only using open apis)
2. implement the DFSClient to DN APIs

Done. Of course 2 is easier said than done and need to look at the details. But, it seems
reasonable to re-factor those APIs to be more standardized and open.

2 need not be done with anything like thrift. could be done with a hadoop rpc layer that's
well defined and can be implemented without reflection etc.

We could also fake 2 by having thriftfs be a bridge to the DN APIs too.

> imeplement DFSClient on top of thriftfs - this may require DFSClient or DN protocol changes
> -------------------------------------------------------------------------------------------
>                 Key: HADOOP-4136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4136
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/thiftfs, dfs, fs
>            Reporter: Pete Wyckoff
>            Priority: Minor
> Open up DFS Protocol to allow non-Hadoop DFS clients to implement reads/writes.  Obviously,
the NN need not be changed because the thriftfs server will serve up the same metadata - ie
it's a bridge to the NN.
> This is useful because if we can do this in Java using more open APIs, we could do it
in C++ or Python or Perl :)
> Doing it in Java first makes sense because we already have the DFSClient - kind of a
proof of concept.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message