Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E9F0919097 for ; Tue, 1 Mar 2016 21:57:18 +0000 (UTC) Received: (qmail 99394 invoked by uid 500); 1 Mar 2016 21:57:18 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 99341 invoked by uid 500); 1 Mar 2016 21:57:18 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 99217 invoked by uid 99); 1 Mar 2016 21:57:18 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2016 21:57:18 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 689282C1F6B for ; Tue, 1 Mar 2016 21:57:18 +0000 (UTC) Date: Tue, 1 Mar 2016 21:57:18 +0000 (UTC) From: "Kihwal Lee (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-7597) DNs should not open new NN connections when webhdfs clients seek MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174484#comment-15174484 ] Kihwal Lee commented on HDFS-7597: ---------------------------------- I kicked the build already : https://builds.apache.org/job/PreCommit-HDFS-Build/14674 I think this is what you started: https://builds.apache.org/job/PreCommit-HDFS-Build/14677 Better kill one to avoid wasting resources. > DNs should not open new NN connections when webhdfs clients seek > ---------------------------------------------------------------- > > Key: HDFS-7597 > URL: https://issues.apache.org/jira/browse/HDFS-7597 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs > Affects Versions: 2.0.0-alpha > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Priority: Critical > Labels: BB2015-05-TBR > Attachments: HDFS-7597.patch, HDFS-7597.patch, HDFS-7597.patch > > > Webhdfs seeks involve closing the current connection, and reissuing a new open request with the new offset. The RPC layer caches connections so the DN keeps a lingering connection open to the NN. Connection caching is in part based on UGI. Although the client used the same token for the new offset request, the UGI is different which forces the DN to open another unnecessary connection to the NN. > A job that performs many seeks will easily crash the NN due to fd exhaustion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)