hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7597) DNs should not open new NN connections when webhdfs clients seek
Date Tue, 01 Mar 2016 21:41:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174459#comment-15174459

Chris Nauroth commented on HDFS-7597:

[~daryn], thank you for the reminder.  It was a surprise, because I honestly thought we had
already closed down on this one.

I remain +1 for the patch, same as several months ago.  I manually triggered a Jenkins run
since it has been a while.  I'll plan on committing this tomorrow barring any objections or
surprises from pre-commit.

> DNs should not open new NN connections when webhdfs clients seek
> ----------------------------------------------------------------
>                 Key: HDFS-7597
>                 URL: https://issues.apache.org/jira/browse/HDFS-7597
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-7597.patch, HDFS-7597.patch, HDFS-7597.patch
> Webhdfs seeks involve closing the current connection, and reissuing a new open request
with the new offset.  The RPC layer caches connections so the DN keeps a lingering connection
open to the NN.  Connection caching is in part based on UGI.  Although the client used the
same token for the new offset request, the UGI is different which forces the DN to open another
unnecessary connection to the NN.
> A job that performs many seeks will easily crash the NN due to fd exhaustion.

This message was sent by Atlassian JIRA

View raw message