hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nitay Joffe (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3754) Support a Thrift Interface to access files/directories in HDFS
Date Tue, 05 Aug 2008 20:38:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620033#action_12620033

Nitay Joffe commented on HADOOP-3754:

This is great work Dhruba!

I worked on a very similar system at Powerset and had a few questions/comments:

1) I wrote a C++ stream using boost::iostreams which used the thrift API underneath so that
you can work with a standard stream yet read/write to hadoop. In order to make this work,
I had to add a seek() method to the IDL. On input streams, I would allow arbitrary seeking.
On output streams I would only allow it to get called with an offset of 0 (no actual seeking)
which boost streams uses to find the current location.

2) I had situations where the client would not close the files appropriately. This meant that
other users would not see the file even though the writing was done because the data did not
appear until the file is closed. To fix this situation, I put a TimerTask on each fd which
would timeout (and close the file) after some period. Whenever an operation was called on
a Handle its TimerTask would reset.

I would attach code but we are currently being merged into Microsoft and in the process of
figuring out how open source contributions will work.


> Support a Thrift Interface to access files/directories in HDFS
> --------------------------------------------------------------
>                 Key: HADOOP-3754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3754
>             Project: Hadoop Core
>          Issue Type: New Feature
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: hadoopthrift2.patch, hadoopthrift3.patch, thrift1.patch
> Thrift is a cross-language RPC framework. It supports automatic code generation for a
variety of languages (Java, C++, python, PHP, etc) It would be nice if HDFS APIs are exposed
though Thirft. It will allow applications written in any programming language to access HDFS.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message