hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API
Date Wed, 23 Sep 2015 22:22:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905406#comment-14905406
] 

Haohui Mai commented on HDFS-8766:
----------------------------------

Thanks for the work.

It looks like that the code needs a lot of clean ups. The code needs to follow the Google
C++ styling guide and be formatted using clang-format.

I think it's important to have several abstractions. There should be C++ classes that correspond
FileSystem / FileHandle as Bob mentioned. 

{code}
+  ssize_t pread(void *buf, size_t nbyte, off_t offset) {
+    auto stat = std::make_shared<std::promise<Status>>();
+    std::future<Status> future(stat->get_future());
+
+    //wrap async call with promise/future to make it blocking
+    size_t readCount = 0;
+    std::string datanode;
+    auto h = [stat, &readCount, &datanode](const Status &s, const std::string
&dn, size_t bytes) {
+      stat->set_value(s);
+      readCount = bytes;
+      datanode = dn;        
+    };
+
+    inputStream->PositionRead(buf, nbyte, offset, bad_datanodes, h);
+
+    //wait for async to finish
+    auto s = future.get();
+
+    if(!s.ok()) {
+      //for now assume this was a down DN, not going to reinvent HDFS-9103 
+      if(s.code() == Status::kResourceUnavailable) {
+        errno = ECOMM;
+        bad_datanodes.insert(datanode);
+      }      
+      //otherwise it was a real error so indicate that something's gone wrong
+      return -1;
+    }
{code}

It should be a generic template to abstract the operations of wrapping async calls to synchronous
calls.

> Implement a libhdfs(3) compatible API
> -------------------------------------
>
>                 Key: HDFS-8766
>                 URL: https://issues.apache.org/jira/browse/HDFS-8766
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: James Clampffer
>            Assignee: James Clampffer
>         Attachments: HDFS-8766.HDFS-8707.000.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in libhdfs and libhdfs3.
 This will make it possible for projects using libhdfs/libhdfs3 to relink against libhdfspp
with minimal changes.
> This also provides a pure C interface that can be linked against projects that aren't
built in C++11 mode for various reasons but use the same compiler.  It also allows many other
programming languages to access libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier for programs
built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message