hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5760) [ATSv2] Create HBase connection only if an app collector is publishing from NM
Date Fri, 21 Oct 2016 19:38:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596122#comment-15596122

Naganarasimha G R commented on YARN-5760:

Thanks [~varun_saxena], for raising it. I had few points to add to this. 
# Well IIUC it might not be just limited to the connection to zookeeper alone but to the RegionServers
too. And also IIRC there is a limit on number of connections which can be opened to it. So
though our assumption of it was not effectively captured in the code.
# Though it was not captured as part of the code our assumption earlier was *# of apps <
 # of nodes*, hence Connection pool (of max size 1 and 0 when not used) would be ideal considering
the previous comment.
# One futuristic case is that we have not visited the system events. Suppose we want to report
NM related system events ( say at given point in time how much resource is allocated and actual
used for a given node, containers running simultaneously at a given point in time) or any
other failures due to which NM goes down, then hope we do not end up in almost existing approach.

> [ATSv2] Create HBase connection only if an app collector is publishing from NM
> ------------------------------------------------------------------------------
>                 Key: YARN-5760
>                 URL: https://issues.apache.org/jira/browse/YARN-5760
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
> Irrespective of NM handling an app or not, we initialize HBaseTimelineWriterImpl in TimelineCollectorManager.
> This in turn calls ConnectionFactory#createConnection to manage connections with HBase.

> But it seems this opens up a connection with Zookeeper (i.e. as soon as NM starts up)
instead of opening connection when atleast one app arrives for publishing and closing it if
no apps are being published from this NM.
> This leads to unnecessary connections to Zookeeper.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message