Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+Vtps7bR2+fXvjAAQo+irP40JoxbxNDb48BhqYsa-xkSh1=Nw@mail.gmail.com>
References: <CAM+GRP94oN=D=MM-PF3rmwkv=jcBuSsSKwag0bkcOhwvrh0fsw@mail.gmail.com>
 <CA+Vtps7bR2+fXvjAAQo+irP40JoxbxNDb48BhqYsa-xkSh1=Nw@mail.gmail.com>
From: Noe Detore <ndetore@minerkasch.com>
Date: Thu, 13 Oct 2016 10:49:30 -0400
Message-ID: <CAM+GRP8aqf6FRFQG-AWFpt9mGdDMBYUqDBoVrZH+s0pfmSN6iQ@mail.gmail.com>
Subject: Re: Lost tablet server lock..SESSION_EXPIRED
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=94eb2c11718096b8c7053ec0371b
archived-at: Thu, 13 Oct 2016 14:49:36 -0000

--94eb2c11718096b8c7053ec0371b
Content-Type: text/plain; charset=UTF-8

Yes, seeing a lot of DEBUG:Upsess. Also seeing
 [server.GarbageCollectionLogger] DEBUG: gc ParNew=64.69(+1.24) secs
ConcurrentMarkSweep=102.51(+0.06) secs
freemem=4,844,821,808(-20,292,780,896) totalmem=25,525,551,104
2016-10-13 11:22:17,963 [zookeeper.ZooLock] DEBUG: event null None
Disconnected

During hotspot seems like a java gc pause is causing zk heart beat to miss
and then expire. Are there recommend java gc configurations?  We are using
native memory. Would trying G1 gc be advised?

Thank you

On Fri, Oct 7, 2016 at 8:23 PM, Jeff Kubina <jeff.kubina@gmail.com> wrote:

> Noe,
>
> Do you have a lot (1000s) of "[tserver.TableServer] DEBUG: UpSess ..."
> messages in your tserver logs prior to the FATAL or "ERROR: Lost tablet
> server lock" error message?
>
> Jeff
>
>
> --
> Jeff Kubina
> 410-988-4436
>
>
> On Fri, Oct 7, 2016 at 10:34 AM, Noe Detore <ndetore@minerkasch.com>
> wrote:
>
>> Any updates on this issue https://issues.apache.org/jira
>> /browse/ACCUMULO-3336 ? I am seeing this behavior using 1.7.2 on one of
>> our clusters. Not seeing on other clusters, but what could be some causes?
>> Swap on server looks good as there is none. Are there particular
>> configurations to adjust?
>>
>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired ...
>> 2016-10-06 23:22:30,633 [zookeeper.DistributedWorkQueue] INFO : Got
>> unexpected zookeeper event: None for ...
>> 2016-10-06 23:22:30,679 [tserver.TabletServer] ERROR: Lost tablet server
>> lock (reason = SESSION_EXPIRED), exiting
>>
>> Thanks
>> Noe
>>
>
>

--94eb2c11718096b8c7053ec0371b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Yes, seeing a lot of DEBUG:Upsess. Also seeing =C2=A0[serv=
er.GarbageCollectionLogger] DEBUG: gc ParNew=3D64.69(+1.24) secs Concurrent=
MarkSweep=3D102.51(+0.06) secs freemem=3D4,844,821,808(-20,292,780,896) tot=
almem=3D25,525,551,104<div>2016-10-13 11:22:17,963 [zookeeper.ZooLock] DEBU=
G: event null None Disconnected</div><div><br></div><div>During hotspot see=
ms like a java gc pause is causing zk heart beat to miss and then expire. A=
re there recommend java gc configurations?=C2=A0 We are using native memory=
. Would trying G1 gc be advised?</div><div><br></div><div>Thank you</div></=
div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Fri, Oct 7=
, 2016 at 8:23 PM, Jeff Kubina <span dir=3D"ltr">&lt;<a href=3D"mailto:jeff=
.kubina@gmail.com" target=3D"_blank">jeff.kubina@gmail.com</a>&gt;</span> w=
rote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde=
r-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div>Noe,<br>=
<br></div>Do you have a lot (1000s) of &quot;[tserver.TableServer] DEBUG: U=
pSess ...&quot; messages in your tserver logs prior to the FATAL or &quot;E=
RROR: Lost tablet server lock&quot; error message?<span class=3D"HOEnZb"><f=
ont color=3D"#888888"><br><br></font></span></div><span class=3D"HOEnZb"><f=
ont color=3D"#888888">Jeff<br><br></font></span></div><div class=3D"gmail_e=
xtra"><span class=3D"HOEnZb"><font color=3D"#888888"><br clear=3D"all"><div=
><div class=3D"m_4274001660853141412gmail_signature" data-smartmail=3D"gmai=
l_signature"><div dir=3D"ltr"><div>--=C2=A0</div>Jeff Kubina<br><a href=3D"=
tel:410-988-4436" value=3D"+14109884436" target=3D"_blank">410-988-4436</a>=
<div><br></div></div></div></div></font></span><div><div class=3D"h5">
<br><div class=3D"gmail_quote">On Fri, Oct 7, 2016 at 10:34 AM, Noe Detore =
<span dir=3D"ltr">&lt;<a href=3D"mailto:ndetore@minerkasch.com" target=3D"_=
blank">ndetore@minerkasch.com</a>&gt;</span> wrote:<br><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_=
extra">Any updates on this issue <a href=3D"https://issues.apache.org/jira/=
browse/ACCUMULO-3336" target=3D"_blank">https://issues.apache.org/jira<wbr>=
/browse/ACCUMULO-3336</a> ? I am seeing this behavior using 1.7.2 on one of=
 our clusters. Not seeing on other clusters, but what could be some causes?=
 Swap on server looks good as there is none. Are there particular configura=
tions to adjust?</div><div class=3D"gmail_extra"><br></div><div class=3D"gm=
ail_extra">org.apache.zookeeper.KeeperExc<wbr>eption$SessionExpiredExceptio=
n<wbr>: KeeperErrorCode =3D Session expired ...=C2=A0</div><div class=3D"gm=
ail_extra">2016-10-06 23:22:30,633 [zookeeper.DistributedWorkQueu<wbr>e] IN=
FO : Got unexpected zookeeper event: None for ...=C2=A0</div><div class=3D"=
gmail_extra">2016-10-06 23:22:30,679 [tserver.TabletServer] ERROR: Lost tab=
let server lock (reason =3D SESSION_EXPIRED), exiting</div><div class=3D"gm=
ail_extra"><br></div><div class=3D"gmail_extra">Thanks</div><span class=3D"=
m_4274001660853141412HOEnZb"><font color=3D"#888888"><div class=3D"gmail_ex=
tra">Noe</div></font></span></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>

--94eb2c11718096b8c7053ec0371b--