hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2346) DataNode should have timeout on socket writes.
Date Wed, 06 Feb 2008 18:19:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566250#action_12566250
] 

Raghu Angadi commented on HADOOP-2346:
--------------------------------------


Java Selector creation seems pretty heavy for a single filedescriptor polling. The current
patch creates one selector for each stream. Looks like each selector takes up 3 fds: 2 for
a pipe (used for {{wakeup()}, I guess) and for epoll().

Even if we are ok with these extra fds for each stream, it still requires users to close the
stream explicitly (to close the selector). Opening and closing new selector each time we wait
might be costly be because of this extra pipe it uses.

If these streams aim to be drop in replacements for Socket's own streams, they should either
clean up  automatically when the channel is closed, or better, should not require a {{close()}}.

Anyone aware of a bare-bones selector or an alternate to a selector? Some thing that is a
thin wrapper over {{poll()}}, without extra set up and extra fds for pipe and epoll?

I think it is still possible to implement these streams with a single global Selector, without
an extra thread. Registering and deregistering a channel each time we want to wait won't be
costly. This way, stream does not need a close().




> DataNode should have timeout on socket writes.
> ----------------------------------------------
>
>                 Key: HADOOP-2346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2346
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.15.1
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>             Fix For: 0.16.1
>
>         Attachments: HADOOP-2346.patch, HADOOP-2346.patch, HADOOP-2346.patch, HADOOP-2346.patch
>
>
> If a client opens a file and stops reading in the middle, DataNode thread writing the
data could be stuck forever. For DataNode sockets we set read timeout but not write timeout.
I think we should add a write(data, timeout) method in IOUtils that assumes it the underlying
FileChannel is non-blocking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message