hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10796) libhdfs++: rationalize ioservice interactions
Date Mon, 05 Dec 2016 17:25:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722783#comment-15722783

James Clampffer commented on HDFS-10796:

Planning to add a few things inline with Bob's original description:
-IoServices will be sharable and thread safe; most likely just take advantage of std::enabled_shared_from_this
-IoService will own it's worker threads so that FileSystems and other async consumers don't
need to specify if they want to default to hardware threads they can also explicitly set to
a number.
-IoService will expose a Post method just like the underlying asio::io_service.  This will
help with HDFS-10241 in places where we currently extract the asio io_service to do posts
for various methods that use async calls for lock scoping.  Should also be useful in tests
when combined with self contained worker threads.
-IoService becomes it's own component with regard to logging.  The service will be able dump
info about thread count and usage statistics which should be handy for sharing between FileSystems.

> libhdfs++: rationalize ioservice interactions
> ---------------------------------------------
>                 Key: HDFS-10796
>                 URL: https://issues.apache.org/jira/browse/HDFS-10796
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Bob Hansen
>            Assignee: James Clampffer
> Firstly, we should be pulling the number of threads from options.io_threads (which should
default to std::thread::hardware_concurrency()).  The library should pass all tests always
with io_threads set to 1 or to <a very high number>
> Secondly, we should have _a_ constructor where the consumer doesn't need to manage the
IOService explicitly, and the FileSystemImpl should create its own internally.
> Since the FileSystem is defined as being for a particular user/identity, there is a valid
use case for the consumer to be constructing many FileSystem instances to represent many authenticated
users in the same process, but want to share resources (notably have a single io_service shared
amongst them all).  In this case, the consumer would want to own the IOService and pass the
same instance to multiple FileSystem instances.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message