hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tarun Parimi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController
Date Thu, 28 Feb 2019 14:38:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780584#comment-16780584
] 

Tarun Parimi commented on YARN-9336:
------------------------------------

Fixed checkstyle issue. I don't think we can add a unit test for this.

> JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController
> ------------------------------------------------------------------------------------------------
>
>                 Key: YARN-9336
>                 URL: https://issues.apache.org/jira/browse/YARN-9336
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: log-aggregation
>    Affects Versions: 2.9.0
>            Reporter: Tarun Parimi
>            Assignee: Tarun Parimi
>            Priority: Major
>         Attachments: YARN-9336.001.patch, YARN-9336.002.patch
>
>
> The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever viewing
a huge log file in JobhistoryServer. This happens only when the below is configured.
> {code:java}
> yarn.log-aggregation.file-formats=IndexedFormat
> yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
> yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
> {code}
> On investigation, I found that the FSDataInputStream is not closed in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
. Since this block is called every time the Jobhistory page displays the logs, CLOSE_WAIT
connections to DataNodes keep on increasing in JobHistoryServer.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message