hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10704) libhdfs++: FileSystem should not take ownership of IOService
Date Sun, 31 Jul 2016 15:11:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401181#comment-15401181
] 

James Clampffer commented on HDFS-10704:
----------------------------------------

What are the main benefits of this?  I have a feeling this will make the implementation a
bit more completed and a bit harder to debug.  What's the current issue preventing multiple
users from using the same filesystem?  Kerberos tokens?  The way I see it:

Pros:
-Can better represent some architectures which is pretty nice
-a few less io_service instances
-user explicitly manages more of the library.
Cons:
-io_service is where a lot of lock contention lives, if someone wants to avoid it they'll
end up using multiple io_services and some structure to track them
-each filesystem is most likely going to track it's worker threads anyway
-big one, as someone who's done a ton of the debugging inside asio, it makes things harder
to debug/fix

I've been thinking for a while that the RpcEngine should have it's own asio::io_service. 
Most of the difficult bugs I've hit manifest as something blowing up right around the io_service:
dangling pointers to std::function callbacks, asio throws an exeption that knocks out a worker
thread, file descriptor closed while another asio::socket object references it, so IMO more
io_services isolates issues.  Performance should be pretty similar in either case; the io_service
is just a wrapper around epoll or select state + some extra state to hold a fd condition->iovec/callback
mapping.  On the other hand I'm all for moving things to requiring users to be smart about
object dependencies so we can start getting rid of shared_ptrs in other places so I'd be happy
if this pushes things in that direction.

> libhdfs++: FileSystem should not take ownership of IOService
> ------------------------------------------------------------
>
>                 Key: HDFS-10704
>                 URL: https://issues.apache.org/jira/browse/HDFS-10704
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Bob Hansen
>
> The ctor for the filesystem currently takes ownership of the IOService passed in.  There
is a valid use case for a single IOService for multiple FileSystems (e.g. for different users).
> If an IOService is passed in, the consumer should be responsible for its lifetime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message