Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 061A6173F1 for ; Fri, 9 Jan 2015 21:53:34 +0000 (UTC) Received: (qmail 49476 invoked by uid 500); 9 Jan 2015 21:53:35 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 49439 invoked by uid 500); 9 Jan 2015 21:53:34 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 49427 invoked by uid 99); 9 Jan 2015 21:53:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jan 2015 21:53:34 +0000 Date: Fri, 9 Jan 2015 21:53:34 +0000 (UTC) From: "Daryn Sharp (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-7597) Clients seeking over webhdfs may crash the NN MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271922#comment-14271922 ] Daryn Sharp commented on HDFS-7597: ----------------------------------- Good question, but synchronizing the whole operation will cause a cache-miss on one lookup to stall all other lookups including those that will be cache hits. While it might be tolerable on the the client-side with minimal impact, I don't think it's worth dragging down the performance of the server-side connection handling. > Clients seeking over webhdfs may crash the NN > --------------------------------------------- > > Key: HDFS-7597 > URL: https://issues.apache.org/jira/browse/HDFS-7597 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs > Affects Versions: 2.0.0-alpha > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Priority: Critical > Attachments: HDFS-7597.patch > > > Webhdfs seeks involve closing the current connection, and reissuing a new open request with the new offset. The RPC layer caches connections so the DN keeps a lingering connection open to the NN. Connection caching is in part based on UGI. Although the client used the same token for the new offset request, the UGI is different which forces the DN to open another unnecessary connection to the NN. > A job that performs many seeks will easily crash the NN due to fd exhaustion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)