hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6695) Investigate using Java 7's nonblocking file I/O in BlockReaderLocal to implement read timeouts
Date Thu, 17 Jul 2014 18:21:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065288#comment-14065288

Colin Patrick McCabe commented on HDFS-6695:

bq. I did not look through AsynchronousfileChannel impl, but per the above link saying: "
This class also defines read and write methods that initiate asynchronous operations, returning
a Future to represent the pending result of the operation. The Future may be used to check
if the operation has completed, wait for its completion, and retrieve the result. ", seems
does the same idea like my pseudo code...

Yes, we could definitely set up a thread pool ourselves, but we don't want to do that because
it would be slow.  Passing large amounts of data between threads can mean moving it between
CPU cores, which really kills performance.  It also introduces latency from the context switches.
 I was hoping that Java7 had support for the kernel's native AIO interface... I guess we'll
have to see.

We might be able to do something interesting by having a thread pool where certain threads
were locked to certain physical CPU cores.  Of course, that requires JNI to pull off, at the
moment, and the overheads of that might wipe out any gain...

> Investigate using Java 7's nonblocking file I/O in BlockReaderLocal to implement read
> ----------------------------------------------------------------------------------------------
>                 Key: HDFS-6695
>                 URL: https://issues.apache.org/jira/browse/HDFS-6695
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Colin Patrick McCabe
> In BlockReaderLocal, the "read" system call could block for a long time if the disk drive
is having problems, or there is a huge amount of I/O contention.  This might cause poor latency
> In the remote block readers, we have implemented a read timeout, but we don't have one
for the local block reader, since {{FileChannel#read}} doesn't support this.  
> Once we move to JDK 7, we should investigate the {{java.nio.file}} nonblocking file I/O
package to see if it could be used to implement read timeouts.

This message was sent by Atlassian JIRA

View raw message