accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Kepner <kep...@ll.mit.edu>
Subject Re: ingest performance oscillations and Xceivers
Date Thu, 03 Jan 2013 22:25:34 GMT
No correlation with compactions.  No queries.

On Thu, Jan 03, 2013 at 11:24:17AM -0500, William Slacum wrote:
> Have you also been tracking compactions? Did you have a query load?
> 
> 
> On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL <
> kepner@ll.mit.edu> wrote:
> 
> > Hmmm, that's interesting, because in the past I didn't see this behavior.
> >  It might be worth having someone look into because it seems to have a 2x
> > impact on sustained ingest.
> >
> > Regards.  -Jeremy
> >
> > On Jan 2, 2013, at 2:23 PM, Keith Turner wrote:
> >
> > > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <kepner@ll.mit.edu> wrote:
> > >> So what mechanism causes the number of Xceivers to increase?
> > >
> > > Its been a while since I looked at the data node source code.   When I
> > > last look at it an Xceiver was just a thread created to handle a
> > > datanode request.   The thread went away after the request was
> > > processed.   So major and minor compactions running would cause more
> > > Xceivers to be created to read and write data.
> > >
> > > Newer datanode code may use a thread pool instead of creating a
> > > thread/xceiver for each request.   I am not sure.
> > >
> > >> I am carefully controlling the number of ingestors and the data isn't
> > varying too much.
> > >> I would expect the number of Xceivers to remain consant.
> > >>
> > >> Regards.  -Jeremy
> > >>
> > >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote:
> > >>> Hey Jeremy,
> > >>>
> > >>> Can you compare the ingest rate to the number of tablets, too?
> > >>>
> > >>> I've found, that if I have 20-80 tablets per server (on similar
> > hardware) I
> > >>> get the best performance.
> > >>>
> > >>> # of Xceivers == number of writers when ingest is the primary target.
> > >>>
> > >>> Also, is this 1.4 or trunk?
> > >>>
> > >>> -Eric
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
> > >>> kepner@ll.mit.edu> wrote:
> > >>>
> > >>>> Accumulo Colleagues,
> > >>>>  I am trying to optimize my ingest into a single node Accumulo
> > instance
> > >>>> running on a 32 core node with 96 GB of RAM.  I am seeing the follow
> > ingest
> > >>>> variations as a I change the number of ingest processes (see
> > attached):
> > >>>>
> > >>>> -------------------------------------
> > >>>> Ingestors, Ingest rate
> > >>>> -------------------------------------
> > >>>> 1, 60K inserts/sec (stable)
> > >>>> 2, 120K inserts/sec (stable)
> > >>>> 3, 60K to 180K inserts/sec
> > >>>> 4, 90K to 220K inserts/sec
> > >>>> 8, 80K to 280K inserts/sec
> > >>>> 12, 80K to 280K inserts/sec
> > >>>> -------------------------------------
> > >>>>
> > >>>> The only thing I can see that correlates with the ingest rate is
the
> > >>>> number of Xceivers.  When the ingest rate is high the number of
> > Xceivers is
> > >>>> usually low.  Likewise, when the ingest rate drops, the number
of
> > Xceivers
> > >>>> usually increases significantly.
> > >>>>
> > >>>> Question: What role to Xceivers play in ingest?
> > >>>>
> > >>>> Request: It would be great to add a plot showing the number of
> > Xceivers
> > >>>> over time to the diagnostics.
> > >>>>
> > >>>> Regards.  -Jeremy
> > >>>>
> > >>>>
> >
> >

Mime
View raw message