hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10796) libhdfs++: rationalize ioservice interactions
Date Fri, 09 Dec 2016 22:01:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736462#comment-15736462
] 

James Clampffer commented on HDFS-10796:
----------------------------------------

Thanks for taking a look at this Xiaowei!

I ran the unpatched HDFS-8707 in docker but changed the number of threads spawned in FileSystemImpl's
constructor to hardware concurrency like this patch patch does and it fails in the exact same
way.

It looks like the minidfscluster bindings in hdfspp_mini_dfs.h fail when doing JNI stuff,
but I don't see anything obvious like thread local state that should be breaking things since
we only hit the JNI from the initiator thread.  I can dig deeper into that stuff but might
be worth pushing into another jira since it looks like the changes here aren't doing anything
new.

> libhdfs++: rationalize ioservice interactions
> ---------------------------------------------
>
>                 Key: HDFS-10796
>                 URL: https://issues.apache.org/jira/browse/HDFS-10796
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Bob Hansen
>            Assignee: James Clampffer
>         Attachments: HDFS-10796.HDFS-8707.000.patch
>
>
> Firstly, we should be pulling the number of threads from options.io_threads (which should
default to std::thread::hardware_concurrency()).  The library should pass all tests always
with io_threads set to 1 or to <a very high number>
> Secondly, we should have _a_ constructor where the consumer doesn't need to manage the
IOService explicitly, and the FileSystemImpl should create its own internally.
> Since the FileSystem is defined as being for a particular user/identity, there is a valid
use case for the consumer to be constructing many FileSystem instances to represent many authenticated
users in the same process, but want to share resources (notably have a single io_service shared
amongst them all).  In this case, the consumer would want to own the IOService and pass the
same instance to multiple FileSystem instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message