hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API
Date Wed, 23 Sep 2015 22:22:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905406#comment-14905406

Haohui Mai commented on HDFS-8766:

Thanks for the work.

It looks like that the code needs a lot of clean ups. The code needs to follow the Google
C++ styling guide and be formatted using clang-format.

I think it's important to have several abstractions. There should be C++ classes that correspond
FileSystem / FileHandle as Bob mentioned. 

+  ssize_t pread(void *buf, size_t nbyte, off_t offset) {
+    auto stat = std::make_shared<std::promise<Status>>();
+    std::future<Status> future(stat->get_future());
+    //wrap async call with promise/future to make it blocking
+    size_t readCount = 0;
+    std::string datanode;
+    auto h = [stat, &readCount, &datanode](const Status &s, const std::string
&dn, size_t bytes) {
+      stat->set_value(s);
+      readCount = bytes;
+      datanode = dn;        
+    };
+    inputStream->PositionRead(buf, nbyte, offset, bad_datanodes, h);
+    //wait for async to finish
+    auto s = future.get();
+    if(!s.ok()) {
+      //for now assume this was a down DN, not going to reinvent HDFS-9103 
+      if(s.code() == Status::kResourceUnavailable) {
+        errno = ECOMM;
+        bad_datanodes.insert(datanode);
+      }      
+      //otherwise it was a real error so indicate that something's gone wrong
+      return -1;
+    }

It should be a generic template to abstract the operations of wrapping async calls to synchronous

> Implement a libhdfs(3) compatible API
> -------------------------------------
>                 Key: HDFS-8766
>                 URL: https://issues.apache.org/jira/browse/HDFS-8766
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: James Clampffer
>            Assignee: James Clampffer
>         Attachments: HDFS-8766.HDFS-8707.000.patch
> Add a synchronous API that is compatible with the hdfs.h header used in libhdfs and libhdfs3.
 This will make it possible for projects using libhdfs/libhdfs3 to relink against libhdfspp
with minimal changes.
> This also provides a pure C interface that can be linked against projects that aren't
built in C++11 mode for various reasons but use the same compiler.  It also allows many other
programming languages to access libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier for programs
built using posix filesystem calls to be modified to access HDFS.

This message was sent by Atlassian JIRA

View raw message