Return-Path: 
 <hdfs-issues-return-129063-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CD8718DC2
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Mon,  6 Jul 2015 22:08:11 +0000 (UTC)
Received: (qmail 19838 invoked by uid 500); 6 Jul 2015 22:08:06 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 19783 invoked by uid 500); 6 Jul 2015 22:08:06 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 19767 invoked by uid 99); 6 Jul 2015 22:08:06 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2015 22:08:06 +0000
Date: Mon, 6 Jul 2015 22:08:06 +0000 (UTC)
From: "Xiaobing Zhou (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12841504.1435623992000.111437.1436220486101@Atlassian.JIRA>
In-Reply-To: <JIRA.12841504.1435623992000@Atlassian.JIRA>
References: <JIRA.12841504.1435623992000@Atlassian.JIRA>
 <JIRA.12841504.1435623992444@arcas>
Subject: [jira] [Commented] (HDFS-8696) Reduce the variances of latency of
 WebHDFS
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615751#comment-14615751 ] 

Xiaobing Zhou commented on HDFS-8696:
-------------------------------------

[~bobhansen] hadoop.http.max.threads only applies to HttpServer2 which is an embedded Jetty server, having no effects on Netty. I agree to use it so as not to cause confusion.


> Reduce the variances of latency of WebHDFS
> ------------------------------------------
>
>                 Key: HDFS-8696
>                 URL: https://issues.apache.org/jira/browse/HDFS-8696
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 2.7.0
>            Reporter: Xiaobing Zhou
>            Assignee: Xiaobing Zhou
>         Attachments: HDFS-8696.1.patch, HDFS-8696.2.patch
>
>
> There is an issue that appears related to the webhdfs server. When making two concurrent requests, the DN will sometimes pause for extended periods (I've seen 1-300 seconds), killing performance and dropping connections. 
> To reproduce: 
> 1. set up a HDFS cluster
> 2. Upload a large file (I was using 10GB). Perform 1-byte reads, writing
> the time out to /tmp/times.txt
> {noformat}
> i=1
> while (true); do 
> echo $i
> let i++
> /usr/bin/time -f %e -o /tmp/times.txt -a curl -s -L -o /dev/null "http://<namenode>:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root&length=1";
> done
> {noformat}
> 3. Watch for 1-byte requests that take more than one second:
> tail -F /tmp/times.txt | grep -E "^[^0]"
> 4. After it has had a chance to warm up, start doing large transfers from
> another shell:
> {noformat}
> i=1
> while (true); do 
> echo $i
> let i++
> (/usr/bin/time -f %e curl -s -L -o /dev/null "http://<namenode>:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root");
> done
> {noformat}
> It's easy to find after a minute or two that small reads will sometimes
> pause for 1-300 seconds. In some extreme cases, it appears that the
> transfers timeout and the DN drops the connection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)