Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 23 May 2016 22:27:13 +0000 (UTC)
From: "Yongjun Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12766514.1420832858000.270717.1464042433184@Atlassian.JIRA>
In-Reply-To: <JIRA.12766514.1420832858000@Atlassian.JIRA>
References: <JIRA.12766514.1420832858000@Atlassian.JIRA> <JIRA.12766514.1420832858614@arcas>
Subject: [jira] [Commented] (HDFS-7597) DNs should not open new NN
 connections when webhdfs clients seek
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 23 May 2016 22:27:20 -0000


    [ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297236#comment-15297236 ] 

Yongjun Zhang commented on HDFS-7597:
-------------------------------------

HI [~daryn], [~cnauroth] and [~xiaobingo],

Thanks a lot for your work here.

Per my read of earlier discussion, it makes sense to me to replace the secure part fix of HDFS-8855 with the patch here (which is more general). However, I agree with Chris that we can commit the patch here and defer any further clean-up to a separate issue. I just kicked off a jenkins build to see if tests are clean.

Thanks.

 
> DNs should not open new NN connections when webhdfs clients seek
> ----------------------------------------------------------------
>
>                 Key: HDFS-7597
>                 URL: https://issues.apache.org/jira/browse/HDFS-7597
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-7597.patch, HDFS-7597.patch, HDFS-7597.patch
>
>
> Webhdfs seeks involve closing the current connection, and reissuing a new open request with the new offset.  The RPC layer caches connections so the DN keeps a lingering connection open to the NN.  Connection caching is in part based on UGI.  Although the client used the same token for the new offset request, the UGI is different which forces the DN to open another unnecessary connection to the NN.
> A job that performs many seeks will easily crash the NN due to fd exhaustion.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org