Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of jimmys.email@gmail.com
 designates 209.85.210.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADxc9BnHpcnCu7B_OoyE9_bvocrEgTCMf7WeN2BEFcN8j8DLOg@mail.gmail.com>
References: 
 <CAG3b0HL2Qbiy4w8JdhQU1QWCXgbWBddXz=F7KJZ95if3QsSnKA@mail.gmail.com>
	<CADxc9BnHpcnCu7B_OoyE9_bvocrEgTCMf7WeN2BEFcN8j8DLOg@mail.gmail.com>
Date: Thu, 4 Apr 2013 15:26:17 -0400
Message-ID: 
 <CAG3b0HKv7WyDocfjG7BJPnwCxBeCWa4ZKWYzqEyEQVYbXgo8qg@mail.gmail.com>
Subject: Re: Increasing Ingest Rate
From: Jimmy Lin <jimmys.email@gmail.com>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=089e013a09cad01bf004d98df10a

--089e013a09cad01bf004d98df10a
Content-Type: text/plain; charset=UTF-8

On Thu, Apr 4, 2013 at 2:25 PM, Eric Newton <eric.newton@gmail.com> wrote:

> Have you pre-split your tablet to spread the load out to all the machines?
> Yes.  We are using splits from loading the whole dataset previously.
> Does the data distribution match your splits?
> Yes.  See above.
> Is the ingest data already sorted (that is, it always writes to the last
> tablet)?
> No.  The data writes to multiple tablets concurrently.  We set up a queue
> parameter and divide the data into multiple queues.
> How much memory and how many threads are you using in your batchwriters?
> I believe we have 16GB of memory for the Java writer with 18 threads
> running per server.
>
> Check the ingest rates on tablet server monitor page and look for hot
> spots.
> There are certain servers that have higher ingest rates, and the server
> that is busiest changes over time, but the overall ingestion rate will not
> go up.
>
>

>
>
> On Thu, Apr 4, 2013 at 2:01 PM, Jimmy Lin <jimmys.email@gmail.com> wrote:
>
>> Hello,
>> I am fairly new to Accumulo and am trying to figure out what is
>> preventing my system from ingesting data at a faster rate. We have 15 nodes
>> running a simple Java program that reads and writes to Accumulo and then
>> indexes some data into Solr. The rate of ingest is not scaling linearly
>> with the number of nodes that we start up. I have tried increasing several
>> parameters including:
>>  - limit of file descriptors in linux
>> - max zookeeper connections
>> - tserver.memory.maps.max
>> - tserver_opts memory size
>> - tserver.mutation_queue.max
>> - tserver.scan.files.open.max
>> - tserver.walog.max.size
>> - tserver.cache.data.size
>> - tserver.cache.index.size
>> - hdfs setting for xceivers
>> No matter what changes we make, we cannot get the ingest rate to go over
>> 100k entries/s and about 6 Mb/s. I know Accumulo should be able to ingest
>> faster than this.
>>  Thanks in advance,
>>
>> Jimmy Lin
>>
>>
>
>

--089e013a09cad01bf004d98df10a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quo=
te">On Thu, Apr 4, 2013 at 2:25 PM, Eric Newton <span dir=3D"ltr">&lt;<a hr=
ef=3D"mailto:eric.newton@gmail.com" target=3D"_blank">eric.newton@gmail.com=
</a>&gt;</span> wrote:<br>
<blockquote style=3D"margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-=
color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class=
=3D"gmail_quote"><div dir=3D"ltr"><div>Have you pre-split your tablet to sp=
read the load out to all the machines?=C2=A0</div>
<div>Yes.=C2=A0 We are using splits from loading the whole dataset previous=
ly.<br></div><div>Does the data distribution match your splits?</div><div>Y=
es.=C2=A0 See above.</div><div>Is the ingest data already sorted (that is, =
it always writes to the last tablet)?</div>

<div>No.=C2=A0 The data writes to multiple tablets concurrently.=C2=A0 We s=
et up a queue parameter and divide the data into multiple queues.<br></div>=
<div>How much memory and how many threads are you using in your batchwriter=
s?</div>
<div>I believe we have 16GB of memory for the Java writer with 18 threads r=
unning per server.</div><div><br></div><div>Check the ingest rates on table=
t server monitor page and look for hot spots.</div><div>There are certain s=
ervers that have higher ingest rates, and the server that is busiest change=
s over time, but the overall ingestion rate will not go up.</div>
<div>=C2=A0</div></div></blockquote><blockquote style=3D"margin:0px 0px 0px=
 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-widt=
h:1px;border-left-style:solid" class=3D"gmail_quote"><div dir=3D"ltr"><div>=
=C2=A0</div>
</div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><b=
r><br><div class=3D"gmail_quote">On Thu, Apr 4, 2013 at 2:01 PM, Jimmy Lin =
<span dir=3D"ltr">&lt;<a href=3D"mailto:jimmys.email@gmail.com" target=3D"_=
blank">jimmys.email@gmail.com</a>&gt;</span> wrote:<br>

<blockquote style=3D"margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-=
color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class=
=3D"gmail_quote"><div dir=3D"ltr"><div>Hello,</div><div> </div><div>I am fa=
irly new to Accumulo and am trying to figure out what is preventing my syst=
em from ingesting data at a faster rate.  We have 15 nodes running a simple=
 Java program that reads and writes to Accumulo and then indexes some data =
into Solr.  The rate of ingest is not scaling linearly with the number of n=
odes that we start up.  I have tried increasing several parameters includin=
g:</div>


<div> </div><div>- limit of file descriptors in linux</div><div>- max zooke=
eper connections</div><div>- tserver.memory.maps.max</div><div>- tserver_op=
ts memory size</div><div>- tserver.mutation_queue.max</div><div>- tserver.s=
can.files.open.max</div>


<div>- tserver.walog.max.size</div><div>- tserver.cache.data.size</div><div=
>- tserver.cache.index.size</div><div>- hdfs setting for xceivers</div><div=
> </div><div>No matter what changes we make, we cannot get the ingest rate =
to go over 100k entries/s and about 6 Mb/s.  I know Accumulo should be able=
 to ingest faster than this.</div>


<div> </div><div>Thanks in advance,</div><span><font color=3D"#888888"><div=
><div>=C2=A0</div><div>Jimmy Lin</div></div><div>=C2=A0</div></font></span>=
</div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>

--089e013a09cad01bf004d98df10a--