accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Slacum <wilhelm.von.cl...@accumulo.net>
Subject Re: ingest performance oscillations and Xceivers
Date Wed, 02 Jan 2013 12:09:13 GMT
How many disks do you have? That can be bottle-necking throughput as the
number of Xceivers is related to the number of resources (threads, sockets:
http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/) used at once
to perform operations.

On Tue, Jan 1, 2013 at 6:45 PM, Eric Newton <eric.newton@gmail.com> wrote:

> Hey Jeremy,
>
> Can you compare the ingest rate to the number of tablets, too?
>
> I've found, that if I have 20-80 tablets per server (on similar hardware) I
> get the best performance.
>
> # of Xceivers == number of writers when ingest is the primary target.
>
> Also, is this 1.4 or trunk?
>
> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
> kepner@ll.mit.edu> wrote:
>
> > Accumulo Colleagues,
> >   I am trying to optimize my ingest into a single node Accumulo instance
> > running on a 32 core node with 96 GB of RAM.  I am seeing the follow
> ingest
> > variations as a I change the number of ingest processes (see attached):
> >
> > -------------------------------------
> > Ingestors, Ingest rate
> > -------------------------------------
> > 1, 60K inserts/sec (stable)
> > 2, 120K inserts/sec (stable)
> > 3, 60K to 180K inserts/sec
> > 4, 90K to 220K inserts/sec
> > 8, 80K to 280K inserts/sec
> > 12, 80K to 280K inserts/sec
> > -------------------------------------
> >
> > The only thing I can see that correlates with the ingest rate is the
> > number of Xceivers.  When the ingest rate is high the number of Xceivers
> is
> > usually low.  Likewise, when the ingest rate drops, the number of
> Xceivers
> > usually increases significantly.
> >
> > Question: What role to Xceivers play in ingest?
> >
> > Request: It would be great to add a plot showing the number of Xceivers
> > over time to the diagnostics.
> >
> > Regards.  -Jeremy
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message