Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0214D10CCF for ; Fri, 9 Jan 2015 20:05:35 +0000 (UTC) Received: (qmail 93208 invoked by uid 500); 9 Jan 2015 20:05:36 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 93161 invoked by uid 500); 9 Jan 2015 20:05:36 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 93144 invoked by uid 99); 9 Jan 2015 20:05:36 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jan 2015 20:05:36 +0000 Date: Fri, 9 Jan 2015 20:05:36 +0000 (UTC) From: "Daryn Sharp (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HDFS-7597) Clients seeking over webhdfs may crash the NN MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-7597: ------------------------------ Attachment: HDFS-7597.patch Use a simple LRU cache on hdfs token identifiers. There's a possible race condition if two connections occur simultaneously when there's no cache entry. The result is 2 connections instead of 1, but the race is tiny and is preferable to an unbounded number of connections. > Clients seeking over webhdfs may crash the NN > --------------------------------------------- > > Key: HDFS-7597 > URL: https://issues.apache.org/jira/browse/HDFS-7597 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs > Affects Versions: 2.0.0-alpha > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Priority: Critical > Attachments: HDFS-7597.patch > > > Webhdfs seeks involve closing the current connection, and reissuing a new open request with the new offset. The RPC layer caches connections so the DN keeps a lingering connection open to the NN. Connection caching is in part based on UGI. Although the client used the same token for the new offset request, the UGI is different which forces the DN to open another unnecessary connection to the NN. > A job that performs many seeks will easily crash the NN due to fd exhaustion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)