Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of texpilot@gmail.com designates
 74.125.83.53 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAJRvFdrOhnBo5eL=tPpKZ5sYH+wuXkTDCgccz7v4XaosOaDjFw@mail.gmail.com>
References: 
 <CAPnhrdv951LOAnH=QemD=cL_HyJMtQeuwES_WUsy4Y52dx7Yhw@mail.gmail.com>
	<CAJRvFdrOhnBo5eL=tPpKZ5sYH+wuXkTDCgccz7v4XaosOaDjFw@mail.gmail.com>
Date: Mon, 28 Oct 2013 17:20:18 -0500
Message-ID: 
 <CAPnhrdtGAqUo-C_tY_pXu-u8OfDA3xUe+iqWeZU8WVCX_=OKXg@mail.gmail.com>
Subject: Re: How to reduce number of entries in memory
From: "Terry P." <texpilot@gmail.com>
To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Content-Type: multipart/alternative; boundary=089e0162837e4ded3804e9d4810d

--089e0162837e4ded3804e9d4810d
Content-Type: text/plain; charset=UTF-8

Thanks for the replies. I was approaching it from a data integrity
perspective, as in wanting it flushed to disk in case of a TabletServer
failure.  Last weekend we saw two TabletServers exit the cluster due to a
network glitch, and wouldn't you know that the 04 node was secondary logger
for the 03 node.

In our case, these entries are hanging around in memory *for hours*, as the
ingest rate is not that high.

Perhaps an hourly flush of the table via the shell to get it out to disk
would be the way to go?


On Mon, Oct 28, 2013 at 4:30 PM, Mike Drob <mdrob@mdrob.com> wrote:

> What are you trying to accomplish by reducing the number of entries in
> memory? A tablet server will not minor compact (flush) until the native map
> fills up, but keeping things in memory isn't really a performance concern.
>
> You can force a one-time minor compaction via the shell using the 'flush'
> command.
>
>
> On Mon, Oct 28, 2013 at 5:19 PM, Terry P. <texpilot@gmail.com> wrote:
>
>> Greetings all,
>> For a growing table that currently from zero to 70 million entries this
>> weekend, I'm seeing 4.4 million entries still in memory, though the client
>> programs are supposed to be flushing their entries.
>>
>> Is there a server-side setting to help reduce the number of entries that
>> are in memory (not yet flushed to disk)?  Our system has fairly light
>> performance requirements, so I'm okay if a tweak may result in reduced
>> ingest performance.
>>
>> Thanks in advance,
>> Terry
>>
>
>

--089e0162837e4ded3804e9d4810d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Thanks for the replies. I was approaching it fro=
m a data integrity perspective, as in wanting it flushed to disk in case of=
 a TabletServer failure.=C2=A0 Last weekend we saw two TabletServers exit t=
he cluster due to a network glitch, and wouldn&#39;t you know that the 04 n=
ode was secondary logger for the 03 node.<br>
<br></div>In our case, these entries are hanging around in memory <i>for ho=
urs</i>, as the ingest rate is not that high.<br><br></div>Perhaps an hourl=
y flush of the table via the shell to get it out to disk would be the way t=
o go?<br>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Mon,=
 Oct 28, 2013 at 4:30 PM, Mike Drob <span dir=3D"ltr">&lt;<a href=3D"mailto=
:mdrob@mdrob.com" target=3D"_blank">mdrob@mdrob.com</a>&gt;</span> wrote:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr"><div>What are you trying to accomplish by reducing the num=
ber of entries in memory? A tablet server will not minor compact (flush) un=
til the native map fills up, but keeping things in memory isn&#39;t really =
a performance concern.<br>

<br></div>You can force a one-time minor compaction via the shell using the=
 &#39;flush&#39; command.<br></div><div class=3D"HOEnZb"><div class=3D"h5">=
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Mon, Oct 2=
8, 2013 at 5:19 PM, Terry P. <span dir=3D"ltr">&lt;<a href=3D"mailto:texpil=
ot@gmail.com" target=3D"_blank">texpilot@gmail.com</a>&gt;</span> wrote:<br=
>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div>Greetings all,<br=
></div>For a growing table that currently from zero to 70 million entries t=
his weekend, I&#39;m seeing 4.4 million entries still in memory, though the=
 client programs are supposed to be flushing their entries.<br>


<br></div><div>Is there a server-side setting to help reduce the number of =
entries that are in memory (not yet flushed to disk)?=C2=A0 Our system has =
fairly light performance requirements, so I&#39;m okay if a tweak may resul=
t in reduced ingest performance.<br>


<br></div><div>Thanks in advance,<br>Terry<br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--089e0162837e4ded3804e9d4810d--