Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: <575205E9.1080508@gmail.com>
References: <CAGUtCHoi_gG5Ogdk=DHGcSqPCu3Wp9sD_7RRpaWFSLgDL7JDyA@mail.gmail.com>
	<575205E9.1080508@gmail.com>
Date: Mon, 6 Jun 2016 15:33:18 -0400
Message-ID: <CAGUtCHr0Y1yrxTkebEX5bNe2j+sqw=qNp3JzWjSTwc-dNAb_Pw@mail.gmail.com>
Subject: Re: Completed 2nd long run of Fluo
From: Keith Turner <keith@deenlo.com>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=001a114929e601d5950534a125b0
archived-at: Mon, 06 Jun 2016 19:33:29 -0000

--001a114929e601d5950534a125b0
Content-Type: text/plain; charset=UTF-8

On Fri, Jun 3, 2016 at 6:34 PM, Josh Elser <josh.elser@gmail.com> wrote:

> Great stuff, Keith! Looking forward to picking it apart in more detail:
>
> bq. In the previous run Fluo worker processes were constantly being killed
> by YARN for exceeding memory limits
>
> I remember stumbling across some (surprising) recommendations which were
> to just completely disable the vmem/pmem monitoring to avoid this. Nice
> that you were actually able to work around it "properly".


I am not quite sure why the workers processes are going over the heap usage
by so much, because no native code should be in use.   One possibility is
thread stacks.   The -Xmx jvm option only controls heap usage.  If there
are lots of threads where the stack has been deep in the past, then those
threads may continue to use a good bit of memory.


>
>
> Keith Turner wrote:
>
>> A 2nd multi-day test of Fluo was completed on EC2 a week or two ago.  I
>> just got around to finishing writing it up. The test was run using
>> Accumulo 1.7.1.
>>
>> http://fluo.io/blog/2016/05/17/webindex-long-run-2
>>
>

--001a114929e601d5950534a125b0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">On Fri, Jun 3, 2016 at 6:34 PM, Josh Elser <span dir=3D"ltr">&lt;<a hre=
f=3D"mailto:josh.elser@gmail.com" target=3D"_blank">josh.elser@gmail.com</a=
>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Great stuff, Keith! L=
ooking forward to picking it apart in more detail:<br>
<br>
bq. In the previous run Fluo worker processes were constantly being killed =
by YARN for exceeding memory limits<br>
<br>
I remember stumbling across some (surprising) recommendations which were to=
 just completely disable the vmem/pmem monitoring to avoid this. Nice that =
you were actually able to work around it &quot;properly&quot;.</blockquote>=
<div><br></div><div>I am not quite sure why the workers processes are going=
 over the heap usage by so much, because no native code should be in use.=
=C2=A0=C2=A0 One possibility is thread stacks.=C2=A0=C2=A0 The -Xmx jvm opt=
ion only controls heap usage.=C2=A0 If there are lots of threads where the =
stack has been deep in the past, then those threads may continue to use a g=
ood bit of memory.<br></div><div>=C2=A0</div><blockquote class=3D"gmail_quo=
te" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"=
><div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
Keith Turner wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
A 2nd multi-day test of Fluo was completed on EC2 a week or two ago.=C2=A0 =
I<br>
just got around to finishing writing it up. The test was run using<br>
Accumulo 1.7.1.<br>
<br>
<a href=3D"http://fluo.io/blog/2016/05/17/webindex-long-run-2" rel=3D"noref=
errer" target=3D"_blank">http://fluo.io/blog/2016/05/17/webindex-long-run-2=
</a><br>
</blockquote>
</div></div></blockquote></div><br></div></div>

--001a114929e601d5950534a125b0--