Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Kepner, Jeremy - 0553 - MITLL" <kepner@ll.mit.edu>
To: "user@accumulo.apache.org" <user@accumulo.apache.org>,
        "dev@accumulo.apache.org" <dev@accumulo.apache.org>
CC: "Kepner, Jeremy - 0553 - MITLL" <kepner@ll.mit.edu>
Date: Wed, 2 Jan 2013 19:25:42 -0500
Subject: Re: ingest performance oscillations and Xceivers
Thread-Topic: ingest performance oscillations and Xceivers
Thread-Index: Ac3pSN6eKBKs/7LjSz6gpHnDXPVEDw==
Message-ID: <7C21BEED-BDBD-40D4-97CF-EF60B776540A@ll.mit.edu>
References: <FB59676C-97D4-4447-A5CD-829B82F8892E@ll.mit.edu>
 <CADxc9BnhBwHHjT183GgKDj-GW-gsrTW6URm5qcEmgKM6txbT2A@mail.gmail.com>
 <20130102191131.GA12538@ll.mit.edu>
 <CAGUtCHrt6q-Mtshs41C7VEW0uHxGwpVDbtd0OKGQp=sq3SP3hw@mail.gmail.com>
In-Reply-To: 
 <CAGUtCHrt6q-Mtshs41C7VEW0uHxGwpVDbtd0OKGQp=sq3SP3hw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hmmm, that's interesting, because in the past I didn't see this behavior.  =
It might be worth having someone look into because it seems to have a 2x im=
pact on sustained ingest.=20

Regards.  -Jeremy

On Jan 2, 2013, at 2:23 PM, Keith Turner wrote:

> On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <kepner@ll.mit.edu> wrote:
>> So what mechanism causes the number of Xceivers to increase?
>=20
> Its been a while since I looked at the data node source code.   When I
> last look at it an Xceiver was just a thread created to handle a
> datanode request.   The thread went away after the request was
> processed.   So major and minor compactions running would cause more
> Xceivers to be created to read and write data.
>=20
> Newer datanode code may use a thread pool instead of creating a
> thread/xceiver for each request.   I am not sure.
>=20
>> I am carefully controlling the number of ingestors and the data isn't va=
rying too much.
>> I would expect the number of Xceivers to remain consant.
>>=20
>> Regards.  -Jeremy
>>=20
>> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote:
>>> Hey Jeremy,
>>>=20
>>> Can you compare the ingest rate to the number of tablets, too?
>>>=20
>>> I've found, that if I have 20-80 tablets per server (on similar hardwar=
e) I
>>> get the best performance.
>>>=20
>>> # of Xceivers =3D=3D number of writers when ingest is the primary targe=
t.
>>>=20
>>> Also, is this 1.4 or trunk?
>>>=20
>>> -Eric
>>>=20
>>>=20
>>>=20
>>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
>>> kepner@ll.mit.edu> wrote:
>>>=20
>>>> Accumulo Colleagues,
>>>>  I am trying to optimize my ingest into a single node Accumulo instanc=
e
>>>> running on a 32 core node with 96 GB of RAM.  I am seeing the follow i=
ngest
>>>> variations as a I change the number of ingest processes (see attached)=
:
>>>>=20
>>>> -------------------------------------
>>>> Ingestors, Ingest rate
>>>> -------------------------------------
>>>> 1, 60K inserts/sec (stable)
>>>> 2, 120K inserts/sec (stable)
>>>> 3, 60K to 180K inserts/sec
>>>> 4, 90K to 220K inserts/sec
>>>> 8, 80K to 280K inserts/sec
>>>> 12, 80K to 280K inserts/sec
>>>> -------------------------------------
>>>>=20
>>>> The only thing I can see that correlates with the ingest rate is the
>>>> number of Xceivers.  When the ingest rate is high the number of Xceive=
rs is
>>>> usually low.  Likewise, when the ingest rate drops, the number of Xcei=
vers
>>>> usually increases significantly.
>>>>=20
>>>> Question: What role to Xceivers play in ingest?
>>>>=20
>>>> Request: It would be great to add a plot showing the number of Xceiver=
s
>>>> over time to the diagnostics.
>>>>=20
>>>> Regards.  -Jeremy
>>>>=20
>>>>=20