hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaobing Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8696) Small reads are blocked by large long running reads
Date Tue, 30 Jun 2015 19:05:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608878#comment-14608878
] 

Xiaobing Zhou commented on HDFS-8696:
-------------------------------------

I investigated many aspects of issue, including disk io queue, NN/DN/client packages,  no
help. it finally boils down to some options we’ve to explicitly set for Netty server.
1. Netty NIO event loop boss/worker threads, by default they are 1 and 2, which are too small
to handle intensive workloads
2. Though it is platform dependent, ChannelOption.SO_BACKLOG, the maximum queue length for
incoming connection, default of which is 50, too small to deal with high concurrency.
3. These two ChannelOption.SO_SNDBUF and ChannelOption.SO_RCVBUF(sending and receiving buffer
size) are the key point to this issue. Performance gains benefit from proper setting significantly.
In this case, up to 30 seconds latency will drop to up to 5 in my test env.
4. ChannelOption.WRITE_BUFFER_LOW_WATER_MARK and ChannelOption.WRITE_BUFFER_HIGH_WATER_MARK(thresholds
to indicate channel is writable or not) also matters to drop up to 5 secs latency to less
than 1 sec.

Right now, I am making all these configurable and try best to tune so as to figure out good
HDFS default values to be shipped out.

> Small reads are blocked by large long running reads
> ---------------------------------------------------
>
>                 Key: HDFS-8696
>                 URL: https://issues.apache.org/jira/browse/HDFS-8696
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 2.6.0
>            Reporter: Xiaobing Zhou
>            Assignee: Xiaobing Zhou
>            Priority: Blocker
>
> There is an issue that appears related to the webhdfs server. When making two concurrent
requests, the DN will sometimes pause for extended periods (I've seen 1-300 seconds), killing
performance and dropping connections. 
> To reproduce: 
> 1. set up a HDFS cluster
> 2. Upload a large file (I was using 10GB). Perform 1-byte reads, writing
> the time out to /tmp/times.txt
> {noformat}
> i=1
> while (true); do 
> echo $i
> let i++
> /usr/bin/time -f %e -o /tmp/times.txt -a curl -s -L -o /dev/null "http://<namenode>:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root&length=1";
> done
> {noformat}
> 3. Watch for 1-byte requests that take more than one second:
> tail -F /tmp/times.txt | grep -E "^[^0]"
> 4. After it has had a chance to warm up, start doing large transfers from
> another shell:
> {noformat}
> i=1
> while (true); do 
> echo $i
> let i++
> (/usr/bin/time -f %e curl -s -L -o /dev/null "http://<namenode>:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root");
> done
> {noformat}
> It's easy to find after a minute or two that small reads will sometimes
> pause for 1-300 seconds. In some extreme cases, it appears that the
> transfers timeout and the DN drops the connection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message