hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API
Date Mon, 19 Oct 2015 20:45:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964024#comment-14964024

James Clampffer commented on HDFS-8766:

Thanks Bob and Haohui for the feedback.

Bob, I'll make those changes first so we can get that landed, then right after do the following:

1)  I'm going to open up another jira for cli utilities in an effort to eventually make a
c++ implementation of the hadoop CLI tools.  I'll turn this stub into a unix-like cat executable
so that it can be used for other things as well.  Then move onto stat and other simple fs
2) I can get rid of the INVALIDATE_MEMBER_PTR macro and use unique pointers, but I'd like
to add a deleter to the smart pointers that nulls the pointer value on destruction.  This
sort of debugging tool would be more useful in the pipeline and block reader implementations
because those are where most of the dangling pointers and references tend to originate.
3) Opened HDFS-9265 for this.  It's more broken than I originally thought which explains all
the double deletes that often manifest as calls to pure virtual methods I've been seeing in
crashes.  Inputstream's state owns a unique_ptr to the reader, and the reader is handing out
shared_ptrs of itself to the continuation pipeline.  So it's double deleting by design rather
than a memory stomp.  Fixing the make_shared_from_this usage alone isn't enough.  I can take
a stab at refactoring and fixing this sometime this week.
4) This is the conventional posix signature, this was intentional because google's guide says
follow existing conventions if they are already well known.
5) Agree with Bob, he's already going to be refactoring a lot so I'll keep everything in the
same place and let him move that.

So the patch will cut out the c_api_test executable (1), add unique_ptrs (2) + get rid of
INVALIDATE_MEMBER_PTR, omit (3) because it's in another jira, not change (4) because it falls
into "exceptions to naming rules", and not change (5) because it's going to be handled in
HDFS-9144.  I'll also get rid of hdfs.h and use the one in the tree.

Haohui, can you think of anything else I'm going to need to change?  I think the current implementation,
once rebased, should be sufficient to get on the development branch so Bob can begin HDFS-9144,
and then I'll submit another patch with the changes I mentioned above tomorrow or Wednesday.

> Implement a libhdfs(3) compatible API
> -------------------------------------
>                 Key: HDFS-8766
>                 URL: https://issues.apache.org/jira/browse/HDFS-8766
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: James Clampffer
>            Assignee: James Clampffer
>         Attachments: HDFS-8766.HDFS-8707.000.patch, HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch,
HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, HDFS-8766.HDFS-8707.005.patch,
> Add a synchronous API that is compatible with the hdfs.h header used in libhdfs and libhdfs3.
 This will make it possible for projects using libhdfs/libhdfs3 to relink against libhdfspp
with minimal changes.
> This also provides a pure C interface that can be linked against projects that aren't
built in C++11 mode for various reasons but use the same compiler.  It also allows many other
programming languages to access libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier for programs
built using posix filesystem calls to be modified to access HDFS.

This message was sent by Atlassian JIRA

View raw message