accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kepner, Jeremy - 0553 - MITLL" <kep...@ll.mit.edu>
Subject Re: ingest performance oscillations and Xceivers
Date Thu, 03 Jan 2013 00:25:42 GMT
Hmmm, that's interesting, because in the past I didn't see this behavior.  It might be worth
having someone look into because it seems to have a 2x impact on sustained ingest. 

Regards.  -Jeremy

On Jan 2, 2013, at 2:23 PM, Keith Turner wrote:

> On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <kepner@ll.mit.edu> wrote:
>> So what mechanism causes the number of Xceivers to increase?
> 
> Its been a while since I looked at the data node source code.   When I
> last look at it an Xceiver was just a thread created to handle a
> datanode request.   The thread went away after the request was
> processed.   So major and minor compactions running would cause more
> Xceivers to be created to read and write data.
> 
> Newer datanode code may use a thread pool instead of creating a
> thread/xceiver for each request.   I am not sure.
> 
>> I am carefully controlling the number of ingestors and the data isn't varying too
much.
>> I would expect the number of Xceivers to remain consant.
>> 
>> Regards.  -Jeremy
>> 
>> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote:
>>> Hey Jeremy,
>>> 
>>> Can you compare the ingest rate to the number of tablets, too?
>>> 
>>> I've found, that if I have 20-80 tablets per server (on similar hardware) I
>>> get the best performance.
>>> 
>>> # of Xceivers == number of writers when ingest is the primary target.
>>> 
>>> Also, is this 1.4 or trunk?
>>> 
>>> -Eric
>>> 
>>> 
>>> 
>>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
>>> kepner@ll.mit.edu> wrote:
>>> 
>>>> Accumulo Colleagues,
>>>>  I am trying to optimize my ingest into a single node Accumulo instance
>>>> running on a 32 core node with 96 GB of RAM.  I am seeing the follow ingest
>>>> variations as a I change the number of ingest processes (see attached):
>>>> 
>>>> -------------------------------------
>>>> Ingestors, Ingest rate
>>>> -------------------------------------
>>>> 1, 60K inserts/sec (stable)
>>>> 2, 120K inserts/sec (stable)
>>>> 3, 60K to 180K inserts/sec
>>>> 4, 90K to 220K inserts/sec
>>>> 8, 80K to 280K inserts/sec
>>>> 12, 80K to 280K inserts/sec
>>>> -------------------------------------
>>>> 
>>>> The only thing I can see that correlates with the ingest rate is the
>>>> number of Xceivers.  When the ingest rate is high the number of Xceivers
is
>>>> usually low.  Likewise, when the ingest rate drops, the number of Xceivers
>>>> usually increases significantly.
>>>> 
>>>> Question: What role to Xceivers play in ingest?
>>>> 
>>>> Request: It would be great to add a plot showing the number of Xceivers
>>>> over time to the diagnostics.
>>>> 
>>>> Regards.  -Jeremy
>>>> 
>>>> 


Mime
View raw message