Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of or.sher1@gmail.com designates
 209.85.214.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <14dfea4e.18e1.14b764a95f9.Coremail.c77_cn@163.com>
References: <3fe78fb9.5182.14b6c6317a4.Coremail.c77_cn@163.com>
	<CAOUOv0Ha+KyU8jfSBoAr2u6hv91pC-AQ6UMEqEo+YcjqZGhn_A@mail.gmail.com>
	<FFDEBE0A-3306-4BFA-A23F-624F72C3A03C@gmail.com>
	<75674ac4.d797.14b71242cbb.Coremail.c77_cn@163.com>
	<6A80D478-1C13-45E9-BEF8-3AC14AAD3CD9@gmail.com>
	<14dfea4e.18e1.14b764a95f9.Coremail.c77_cn@163.com>
Date: Wed, 11 Feb 2015 09:36:45 +0200
Message-ID: 
 <CAGA8AHap+rd7vcMY8xK2CA5tN5K1CP_oApHau+w9y1Lwhs-edw@mail.gmail.com>
Subject: Re: Re: Stopping ntpd signals SIGTERM, then causes namenode exit
From: Or Sher <or.sher1@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a114034e0be3a9e050ecb1036

--001a114034e0be3a9e050ecb1036
Content-Type: text/plain; charset=UTF-8

I'm not sure it's related but I encountered a similar issue a few months
ago.
In my case, it was an "at" command sending a kill signal to the at daemon
with it's correct pid.
Somehow, once in a while this signal got to Cassandra process (Java as
well) and killed it.
After some time of investigation I assumed this have to be a kernel bug or
something and I've opened a ticket for CentOS -
http://bugs.centos.org/view.php?id=7539 which no body is really looking at
:)
You can read there how I tried to tackle it.
Bottom line, we've changed the at scheduler to a different implementation
and we don't get this issue any more.

HTH,
Or.


On Wed, Feb 11, 2015 at 3:39 AM, David chen <c77_cn@163.com> wrote:

> The command 'service ntpd stop' could be triggered around 14:00.
> Because the crontab was set as follows:
> 0 * * * * sh sync.sh
> The script contains the following command:
> #!/bin/bash
> service ntpd stop
> ntpdate 192.168.0.1 #it's a valid ntpd server in LAN
> service ntpd start
> chkconfig ntpd on
>
> Found the following fragment in /var/log/message:
> Jan  7 14:00:01 host1 ntpd[32101]: ntpd exiting on signal 15
> Jan  7 13:59:59 host1 ntpd[44764]: ntpd 4.2.4p8@1.1612-o Fri Feb 22
> 11:23:27 UTC 2013 (1)
> Jan  7 13:59:59 host1 ntpd[44765]: precision = 0.143 usec
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on interface #0 wildcard,
> 0.0.0.0#123 Disabled
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on interface #1 wildcard,
> ::#123 Disabled
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on interface #2 lo, ::1#123
> Enabled
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on interface #3 em2,
> fe80::ca1f:66ff:fee1:eed#123 Enabled
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on interface #4 lo,
> 127.0.0.1#123 Enabled
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on interface #5 em2,
> 192.168.1.151#123 Enabled
> Jan  7 13:59:59 host1 ntpd[44765]: Listening on routing socket on fd #22
> for interface updates
> Jan  7 13:59:59 host1 ntpd[44765]: kernel time sync status 2040
> Jan  7 13:59:59 host1 ntpd[44765]: frequency initialized 499.399 PPM from
> /var/lib/ntp/drift
> Jan  7 14:00:01 host1 ntpd_initres[32103]: parent died before we finished,
> exiting
> Jan  7 14:04:17 host1 ntpd[44765]: synchronized to 192.168.0.191, stratum 2
> Jan  7 14:04:17 host1 ntpd[44765]: kernel time sync status change 2001
> Jan  7 14:26:02 host1 snmpd[4842]: Received TERM or STOP signal...
>  shutting down...
> Jan  7 14:26:02 host1 kernel: netlink: 12 bytes leftover after parsing
> attributes.
> Jan  7 14:26:02 host1 snmpd[45667]: NET-SNMP version 5.5
> Jan  7 14:52:48 host1 ntpd[44765]: no servers reachable
>
> It looks likely the command 'service ntpd stop' send the SIGTERM signal.
> The above clue 'ntpd[32101]' indicates that the ntpd process PID is 32101,
> inspect NameNode log, i found that the NameNode process PID was not
> identical with ntpd.
> So i wonder why Namenode process can received the signal?
>


-- 
Or Sher

--001a114034e0be3a9e050ecb1036
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I&#39;m not sure it&#39;s related but I encountered a simi=
lar issue a few months ago.<div>In my case, it was an &quot;at&quot; comman=
d sending a kill signal to the at daemon with it&#39;s correct pid.</div><d=
iv>Somehow, once in a while this signal got to Cassandra process (Java as w=
ell) and killed it.</div><div>After some time of investigation I assumed th=
is have to be a kernel bug or something and I&#39;ve opened a ticket for Ce=
ntOS -=C2=A0<a href=3D"http://bugs.centos.org/view.php?id=3D7539">http://bu=
gs.centos.org/view.php?id=3D7539</a>=C2=A0which no body is really looking a=
t :)</div><div>You can read there how I tried to tackle it.</div><div>Botto=
m line, we&#39;ve changed the at scheduler to a different implementation an=
d we don&#39;t get this issue any more.</div><div><br></div><div>HTH,</div>=
<div>Or.</div><div><br></div></div><div class=3D"gmail_extra"><br><div clas=
s=3D"gmail_quote">On Wed, Feb 11, 2015 at 3:39 AM, David chen <span dir=3D"=
ltr">&lt;<a href=3D"mailto:c77_cn@163.com" target=3D"_blank">c77_cn@163.com=
</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin=
:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style=3D"line=
-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>The comman=
d &#39;service ntpd stop&#39; could be triggered around 14:00.</div><div>Be=
cause the crontab was set as follows:</div><div>0 * * * * sh sync.sh</div><=
div>The script contains the following command:</div><span class=3D""><div>#=
!/bin/bash</div><div>service ntpd stop</div><div>ntpdate 192.168.0.1 #it=
9;s a valid ntpd server in LAN</div><div>service ntpd start</div><div>chkco=
nfig ntpd on</div><div><br></div></span><div>Found the following fragment i=
n /var/log/message:=C2=A0</div><span class=3D""><div>Jan =C2=A07 14:00:01 h=
ost1 ntpd[32101]: ntpd exiting on signal 15</div><div>Jan =C2=A07 13:59:59 =
host1 ntpd[44764]: ntpd 4.2.4p8@1.1612-o Fri Feb 22 11:23:27 UTC 2013 (1)</=
div><div>Jan =C2=A07 13:59:59 host1 ntpd[44765]: precision =3D 0.143 usec</=
div><div>Jan =C2=A07 13:59:59 host1 ntpd[44765]: Listening on interface #0 =
wildcard, 0.0.0.0#123 Disabled</div><div>Jan =C2=A07 13:59:59 host1 ntpd[44=
765]: Listening on interface #1 wildcard, ::#123 Disabled</div><div>Jan =C2=
=A07 13:59:59 host1 ntpd[44765]: Listening on interface #2 lo, ::1#123 Enab=
led</div><div>Jan =C2=A07 13:59:59 host1 ntpd[44765]: Listening on interfac=
e #3 em2, fe80::ca1f:66ff:fee1:eed#123 Enabled</div><div>Jan =C2=A07 13:59:=
59 host1 ntpd[44765]: Listening on interface #4 lo, 127.0.0.1#123 Enabled</=
div><div>Jan =C2=A07 13:59:59 host1 ntpd[44765]: Listening on interface #5 =
em2, 192.168.1.151#123 Enabled</div><div>Jan =C2=A07 13:59:59 host1 ntpd[44=
765]: Listening on routing socket on fd #22 for interface updates</div><div=
>Jan =C2=A07 13:59:59 host1 ntpd[44765]: kernel time sync status 2040</div>=
<div>Jan =C2=A07 13:59:59 host1 ntpd[44765]: frequency initialized 499.399 =
PPM from /var/lib/ntp/drift</div><div>Jan =C2=A07 14:00:01 host1 ntpd_initr=
es[32103]: parent died before we finished, exiting</div><div>Jan =C2=A07 14=
:04:17 host1 ntpd[44765]: synchronized to 192.168.0.191, stratum 2</div><di=
v>Jan =C2=A07 14:04:17 host1 ntpd[44765]: kernel time sync status change 20=
01</div><div>Jan =C2=A07 14:26:02 host1 snmpd[4842]: Received TERM or STOP =
signal... =C2=A0shutting down...</div><div>Jan =C2=A07 14:26:02 host1 kerne=
l: netlink: 12 bytes leftover after parsing attributes.</div><div>Jan =C2=
=A07 14:26:02 host1 snmpd[45667]: NET-SNMP version 5.5</div><div>Jan =C2=A0=
7 14:52:48 host1 ntpd[44765]: no servers reachable</div><div><br></div></sp=
an><div>It looks likely the command &#39;service ntpd stop&#39; send the SI=
GTERM signal. The above clue &#39;ntpd[32101]&#39; indicates that the ntpd =
process PID is 32101, inspect NameNode log, i found that the NameNode proce=
ss PID was not identical with ntpd.=C2=A0</div><div>So i wonder why Namenod=
e process can received the signal?</div></div></blockquote></div><br><br cl=
ear=3D"all"><div><br></div>-- <br><div class=3D"gmail_signature">Or Sher</d=
iv>
</div>

--001a114034e0be3a9e050ecb1036--