Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGFNOZRg16hf+5jLy1o4A4AzzHFZYtjxpJWNFoHd-aknDXnLuQ@mail.gmail.com>
References: 
 <CAGFNOZRg16hf+5jLy1o4A4AzzHFZYtjxpJWNFoHd-aknDXnLuQ@mail.gmail.com>
Date: Tue, 8 Apr 2014 17:35:51 -0400
Message-ID: 
 <CAPMpPc7BYmkSMjPE5-k_hsTHEi4Q1AueSCRR2SeQpXUSyzukUw@mail.gmail.com>
Subject: Re: Advice on increasing ingest rate
From: Adam Fuchs <afuchs@apache.org>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=047d7b86de40a1258504f68ec4b6

--047d7b86de40a1258504f68ec4b6
Content-Type: text/plain; charset=ISO-8859-1

MIke,

What version of Accumulo are you using, how many tablets do you have, and
how many threads are you using for minor and major compaction pools? Also,
how big are the keys and values that you are using?

Here are a few settings that may help you:
1. WAL replication factor (tserver.wal.replication). This defaults to 3
replicas (the HDFS default), but if you set it to 2 it will give you a
performance boost without a huge hit to reliability.
2. Ingest buffer size (tserver.memory.maps.max), also known as the
in-memory map size. Increasing this generally improves the efficiency of
minor compactions and reduces the number of major compactions that will be
required down the line. 4-8 GB is not unreasonable.
3. Make sure your WAL settings are such that the size of a log
(tserver.walog.max.size) multiplied by the number of active logs
(table.compaction.minor.logs.threshold) is greater than the in-memory map
size. You probably want to accomplish this by bumping up the number of
active logs.
4. Increase the buffer size on the BatchWriter that the clients use. This
can be done with the setBatchWriterOptions method on the
AccumuloOutputFormat.

Cheers,
Adam


On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mike@piragua.com> wrote:

> Hello,
>
> We have an ingest process that operates via Map Reduce, processing a large
> set of XML files and  inserting mutations based on that data into a set of
> tables.
>
> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
> get 400k inserts per second with 20 mapper tasks running concurrently.
>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
> effect (besides causing a little more backup in compactions).
>
> I've increased the table.compaction.major.ratio and increased the number
> of concurrent allowed compactions for both min and max compaction but each
> of those only had negligible impact on ingest rates.
>
> Any advice on other settings I can tweak to get things to move more
> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
> where we should consider generating r files like the bulk ingest example?
>
> Thanks in advance for any advice.
>
> Mike
>

--047d7b86de40a1258504f68ec4b6
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">MIke,<input name=3D"virtru-metadata" type=3D"hidden" value=
=3D"{&quot;email-policy&quot;:{&quot;state&quot;:&quot;closed&quot;,&quot;e=
xpirationUnit&quot;:&quot;days&quot;,&quot;disableCopyPaste&quot;:false,&qu=
ot;disablePrint&quot;:false,&quot;disableForwarding&quot;:false,&quot;expir=
es&quot;:false},&quot;attachments&quot;:{}}"><div>
<br></div><div>What version of Accumulo are you using, how many tablets do =
you have, and how many threads are you using for minor and major compaction=
 pools? Also, how big are the keys and values that you are using?</div>
<div><br></div><div>Here are a few settings that may help you:</div><div>1.=
 WAL replication factor (tserver.wal.replication). This defaults to 3 repli=
cas (the HDFS default), but if you set it to 2 it will give you a performan=
ce boost without a huge hit to reliability.</div>
<div>2. Ingest buffer size (tserver.memory.maps.max), also known as the in-=
memory map size. Increasing this generally improves the efficiency of minor=
 compactions and reduces the number of major compactions that will be requi=
red down the line. 4-8 GB is not unreasonable.</div>
<div>3. Make sure your WAL settings are such that the size of a log (tserve=
r.walog.max.size) multiplied by the number of active logs (table.compaction=
.minor.logs.threshold) is greater than the in-memory map size. You probably=
 want to accomplish this by bumping up the number of active logs.</div>
<div>4. Increase the buffer size on the BatchWriter that the clients use. T=
his can be done with the setBatchWriterOptions method on the AccumuloOutput=
Format.</div><div><br></div><div>Cheers,</div><div>Adam</div><div><br></div=
>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Tue,=
 Apr 8, 2014 at 4:47 PM, Mike Hugo <span dir=3D"ltr">&lt;<a href=3D"mailto:=
mike@piragua.com" target=3D"_blank">mike@piragua.com</a>&gt;</span> wrote:<=
br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr">Hello,<div><br></div><div>We have an ingest process that o=
perates via Map Reduce, processing a large set of XML files and =A0insertin=
g mutations based on that data into a set of tables.</div><div><br></div><d=
iv>


On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I get=
 400k inserts per second with 20 mapper tasks running concurrently. =A0Incr=
easing the number of concurrent mapper tasks to 40 doesn&#39;t have any eff=
ect (besides causing a little more backup in compactions).</div>


<div><br></div><div>I&#39;ve increased the table.compaction.major.ratio and=
 increased the number of concurrent allowed compactions for both min and ma=
x compaction but each of those only had negligible impact on ingest rates.<=
/div>


<div><br></div><div>Any advice on other settings I can tweak to get things =
to move more quickly? =A0Or is 400k/second a reasonable ingest rate? =A0Are=
 we at a point where we should consider generating r files like the bulk in=
gest example?</div>


<div><br></div><div>Thanks in advance for any advice.</div><span class=3D"H=
OEnZb"><font color=3D"#888888"><div><br></div><div>Mike</div></font></span>=
</div>
</blockquote></div><br></div>

--047d7b86de40a1258504f68ec4b6--