hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5760) [ATSv2] Create HBase connection only if an app collector is publishing from NM
Date Fri, 21 Oct 2016 20:12:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596237#comment-15596237
] 

Varun Saxena commented on YARN-5760:
------------------------------------

bq. Well IIUC it might not be just limited to the connection to zookeeper alone but to the
RegionServers too.
As per my tests, only connection with Zookeeper is opened.
I believe connections to region servers are not opened till we start writing or reading data
i.e. when an application is active.

There is some work going on for off-application clients which can be used to publish system
events or events from applications like Oozie i.e. entities outside scope of app.
Yes, design for that needs to take care of not opening too many connections to Region Servers.

> [ATSv2] Create HBase connection only if an app collector is publishing from NM
> ------------------------------------------------------------------------------
>
>                 Key: YARN-5760
>                 URL: https://issues.apache.org/jira/browse/YARN-5760
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>
> Irrespective of NM handling an app or not, we initialize HBaseTimelineWriterImpl in TimelineCollectorManager.
> This in turn calls ConnectionFactory#createConnection to manage connections with HBase.

> But it seems this opens up a connection with Zookeeper (i.e. as soon as NM starts up)
instead of opening connection when atleast one app arrives for publishing and closing it if
no apps are being published from this NM.
> This leads to unnecessary connections to Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message