Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-dev@hadoop.apache.org
Subject: Re: [Timeline V2 branch] Latest timeline v2 and SMP problem
MIME-Version: 1.0
From: Li Lu <llu@hortonworks.com>
To: "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
CC: "Naganarasimha G R (Naga)" <garlanaganarasimha@huawei.com>, Varun Saxena
	<vsaxena.varun@gmail.com>, Sangjin Lee <sjlee@twitter.com>, Junping Du
	<jdu@hortonworks.com>, Vrushali Channapattan <vrushali@twitter.com>, "Joep
 Rottinghuis" <joep@twitter.com>
Thread-Topic: [Timeline V2 branch] Latest timeline v2 and SMP problem
Thread-Index: 
 AQHRNtlubQ/H9TgovkCanXYbyya8vp7L1Z6AgAANuQCAACULgIAABaKAgADgIwCAAApEAA==
Date: Tue, 15 Dec 2015 19:37:17 +0000
Message-ID: <CD46F120-08A0-48AC-9F87-54551B7DD05A@hortonworks.com>
References: <4335BFA2-D106-4AC5-8C21-9D8AB2F39516@hortonworks.com>
 <CADbBEnugXOT5APgL2sybcmT2kuW-VnRXO4cHA-ffhkMiZrs_Ng@mail.gmail.com>
 <0FE709BE-FC67-43B4-964D-F6F8AF7D4248@hortonworks.com>
 <CANOghCXm8TOR3esAm69ifNWUKE9PYj2WBcRrexSQDEWx_VuJkQ@mail.gmail.com>
 <AD354F56741A1B47882A625909A59C692BE55825@SZXEML505-MBX.china.huawei.com>
 <13B139A9-AF38-4197-9AD4-5309C90BA121@hortonworks.com>
In-Reply-To: <13B139A9-AF38-4197-9AD4-5309C90BA121@hortonworks.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [192.175.27.10]
x-source-routing-agent: Processed
Content-Type: multipart/alternative;
	boundary="_000_CD46F12008A048AC9F8754551B7DD05Ahortonworkscom_"

--_000_CD46F12008A048AC9F8754551B7DD05Ahortonworkscom_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

OK I fixed this and verified locally. Patch is posted to YARN-4460. Patch i=
s simple so a quick review is much appreciated. Anyone hitting this problem=
 is more than welcome to verify. Thanks!

Li Lu

On Dec 15, 2015, at 11:00, Li Lu <llu@hortonworks.com<mailto:llu@hortonwork=
s.com>> wrote:

Thanks Varun and Naga! I verified locally that the V2 publisher introduced =
in YARN-4129 caused this problem. I=92ll open a JIRA and post a quick fix r=
ight away. Thanks for the information!

Li Lu

On Dec 14, 2015, at 21:38, Naganarasimha G R (Naga) <garlanaganarasimha@hua=
wei.com<mailto:garlanaganarasimha@huawei.com><mailto:garlanaganarasimha@hua=
wei.com>> wrote:

Hi Varun & Li,

Yes Varun most possible reason would be what you mentioned and it has to be=
 done in serviceInit which is taken care in V1 Publisher but missed in V2 P=
ublisher.
Entire logic present in serviceStart of V2Publisher should be moved to serv=
iceInit.

But was wondering for which event/entity ? was it in RM Recover mode ?

Regards,
+ Naga

________________________________
From: Varun Saxena [vsaxena.varun@gmail.com<mailto:vsaxena.varun@gmail.com>=
<mailto:vsaxena.varun@gmail.com>]
Sent: Tuesday, December 15, 2015 10:48
To: Li Lu
Cc: yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org><mailto:ya=
rn-dev@hadoop.apache.org>; Sangjin Lee; Junping Du; Vrushali Channapattan; =
Joep Rottinghuis; Naganarasimha G R (Naga)
Subject: Re: [Timeline V2 branch] Latest timeline v2 and SMP problem

Hi Li,

This is because we are registering the event in serviceStart() instead of s=
erviceInit().
As SMP is the last service in the list, its started right in the end i.e. e=
ven after all the RPCs', UI related stuff.

This can cause an app flow to start before the SMP/V2Publisher service has =
even started. This is what causes the issue.

You want to raise JIRA for this issue or should I ? I can handle it.

Regards,
Varun Saxena.

On Tue, Dec 15, 2015 at 8:35 AM, Li Lu <llu@hortonworks.com<mailto:llu@hort=
onworks.com><mailto:llu@hortonworks.com>> wrote:
Thanks Sangjin. I=92ll keep tracing this. Meanwhile, if anybody has reprodu=
ced the problem, please feel free to let me know. Thanks!

Li Lu

On Dec 14, 2015, at 18:16, Sangjin Lee <sjlee0@gmail.com<mailto:sjlee0@gmai=
l.com><mailto:sjlee0@gmail.com>> wrote:

Can you bisect the commits to see if you can isolate which commit
introduced the issue?

On Mon, Dec 14, 2015 at 5:39 PM, Li Lu <llu@hortonworks.com<mailto:llu@hort=
onworks.com><mailto:llu@hortonworks.com>> wrote:

Hi YARN developers working on Timeline v2 (YARN-2928) branch,

I just realized I=92ve accidentally turned off SMP for my local Timeline v2
build. After I turned yarn.system-metrics-publisher.enabled back on, the RM
fails to start with the following FATAL message:

2015-12-14 17:27:54,125 INFO  ipc.Server (Server.java:run(797)) - IPC
Server listener on 8033: starting
2015-12-14 17:27:54,127 FATAL event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(189)) - Error in dispatcher thread true
java.lang.Exception: No handler for registered for class
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetrics=
Publi
sher$SystemMetricsEventType
      at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:=
185)
      at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109=
)
      at java.lang.Thread.run(Thread.java:745)
2015-12-14 17:27:54,127 INFO  event.AsyncDispatcher
(AsyncDispatcher.java:register(208)) - Registering class
org.apache.hadoop.yarn.serve
r.resourcemanager.metrics.AbstractSystemMetricsPublisher$SystemMetricsEvent=
Type
for class org.apache.hadoop.yarn.server.resourcemanager.m
etrics.TimelineServiceV2Publisher$TimelineV2EventHandler

Interestingly, we=92re registering this class to timeline v2 handler in the
next line of log. I=92m wondering if this is caused by some of my missing
configs, or a newly introduced issue? Has anybody on feature-YARN-2928
branch noticed this issue? Thanks!

Li Lu


--_000_CD46F12008A048AC9F8754551B7DD05Ahortonworkscom_--