accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Supun Kamburugamuva <supu...@gmail.com>
Subject Re: GSOC: Monitor Improvements
Date Mon, 22 Apr 2013 15:27:44 GMT
Thank you all for the valuable input. I'll start writing the proposal. I
really like to contribute to Accumulo and would like to take on the RRDTool
proposal by Eric after the summer. Hopefully I'll have time.

Thanks,
Supun..


On Mon, Apr 22, 2013 at 11:17 AM, Eric Newton <eric.newton@gmail.com> wrote:

> I would do something simpler: just have a Mock collector which does no JMX,
> it just makes up numbers, which could be substituted for testing.
>
> -Eric
>
>
>
> On Mon, Apr 22, 2013 at 11:04 AM, Supun Kamburugamuva <supun06@gmail.com
> >wrote:
>
> > That sounds interesting. To clarify the requirement, we can have a
> process
> > that exposes the same JMX mbeans as the the real server and monitor can
> > plug in to this process.
> >
> > Thanks,
> > Supun..
> >
> >
> > On Mon, Apr 22, 2013 at 10:57 AM, Josh Elser <josh.elser@gmail.com>
> wrote:
> >
> > > That would be pretty sweet, actually. Potentially parallel to what you
> > > want to do, Supun, but cool nonetheless.
> > >
> > > I could see a lot of benefit by having some process that could emulate
> > the
> > > output from a non-trivially-sized Accumulo cluster on a single box.
> > >
> > >
> > > On 4/22/13 10:43 AM, Eric Newton wrote:
> > >
> > >> You could mock the stats collection.
> > >>
> > >> -Eric
> > >>
> > >>
> > >> On Mon, Apr 22, 2013 at 10:41 AM, David Medinets
> > >> <david.medinets@gmail.com>**wrote:
> > >>
> > >>  The average developer probably can't access a large cluster with
> > hundred
> > >>> of
> > >>> nodes. Is there a way to simulate this?
> > >>>
> > >>>
> > >>> On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <eric.newton@gmail.com>
> > >>> wrote:
> > >>>
> > >>>  Another thing to consider is scale.  On large clusters (many
> hundreds
> > of
> > >>>> nodes), more data is not helpful for visualization.  Instead,
> > summaries,
> > >>>> averages and outliers are important.
> > >>>>
> > >>>> For example, if one node is consistently slow, it is better to
know
> > that
> > >>>> than to see one graph with low numbers in a sea of graphs.
> > >>>>
> > >>>> If the monitor collects information using JMX, collection time
for
> > each
> > >>>> node would be a good thing to know, too.
> > >>>>
> > >>>> -Eric
> > >>>>
> > >>>>
> > >>>> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <josh.elser@gmail.com>
> > >>>>
> > >>> wrote:
> > >>>
> > >>>> Supun,
> > >>>>>
> > >>>>> Yup, very much so. Having a way to consume any and all metrics
via
> > JMX
> > >>>>> would simplify things for any consumers (internal or external).
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
> > >>>>>
> > >>>>>  Hi Josh,
> > >>>>>>
> > >>>>>> Thanks for the suggestions. I'll incorporate these to the
> proposal.
> > >>>>>>
> > >>>>>> Another area I would like to work is on JMX. There is a
Jira that
> > says
> > >>>>>>
> > >>>>> to
> > >>>>
> > >>>>> replace the Monitor calls from Thrift to JMX (Accumulo 694).
Do you
> > >>>>>>
> > >>>>> think
> > >>>>
> > >>>>> this is a good addition to the Monitor?
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Supun..
> > >>>>>>
> > >>>>>>
> > >>>>>> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <josh.elser@gmail.com
> >
> > >>>>>>
> > >>>>> wrote:
> > >>>>
> > >>>>>   Supun,
> > >>>>>>
> > >>>>>>> Looks good! Can I make some suggestions/comments?
> > >>>>>>>
> > >>>>>>> For: "Per table plots: ACCUMULO-594", I'd also like
to see minor
> > >>>>>>> compactions, major compactions, index cache hit rate,
and data
> > cache
> > >>>>>>>
> > >>>>>> hit
> > >>>>
> > >>>>> rate per table (same graphs that are displayed system-wide
when you
> > >>>>>>>
> > >>>>>> visit
> > >>>>
> > >>>>> http://${MONITOR_HOST}:50095/.
> > >>>>>>>
> > >>>>>>> For "Per tablet [server] plots", it would be neat if
you could
> also
> > >>>>>>> extract some general statistics like top N least performing,
top
> N
> > >>>>>>> highest
> > >>>>>>> performing, etc. tablet servers. Ideally, this could
correlate
> with
> > >>>>>>> servers
> > >>>>>>> that may be having problems :).
> > >>>>>>>
> > >>>>>>> Do you see these proposed changes as being sufficient
for 3-4
> > months
> > >>>>>>>
> > >>>>>> of
> > >>>
> > >>>>  40hrs/week work? If you plan to really dig into these changes
> > >>>>>>>
> > >>>>>> (perhaps
> > >>>
> > >>>>  reworking components of the monitor itself), I could perhaps see
> > >>>>>>>
> > >>>>>> this.
> > >>>
> > >>>> Do
> > >>>>
> > >>>>> you have any ideas for more lofty goals that you could pursue
as
> > >>>>>>>
> > >>>>>> well?
> > >>>
> > >>>> I
> > >>>>
> > >>>>> don't want you/us to get one month into things and see you
complete
> > >>>>>>> everything we initially planned to accomplish :)
> > >>>>>>>
> > >>>>>>> - Josh
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 04/21/2013 10:37 AM, Supun Kamburugamuva wrote:
> > >>>>>>>
> > >>>>>>>   Hi all,
> > >>>>>>>
> > >>>>>>>> I would like to start writing the proposal for
the GSoc. I've
> put
> > >>>>>>>> together
> > >>>>>>>> some initial high level goals of the project. Please
let me know
> > >>>>>>>>
> > >>>>>>> what
> > >>>
> > >>>> I
> > >>>>
> > >>>>>  can
> > >>>>>>>> improve.
> > >>>>>>>>
> > >>>>>>>> Per table plots: Accumulo 594
> > >>>>>>>> ---------------------
> > >>>>>>>>
> > >>>>>>>> The goal of this is to display plots that explains
the various
> > >>>>>>>> activtities
> > >>>>>>>> that happens per table. When we go to the tables
page of the
> > monitor
> > >>>>>>>>
> > >>>>>>> and
> > >>>>
> > >>>>>  go
> > >>>>>>>> to a specific table it displays some information
in a table
> > format.
> > >>>>>>>>
> > >>>>>>> We
> > >>>
> > >>>>  can
> > >>>>>>>> argument this information by showing graphs for
> > >>>>>>>>
> > >>>>>>>> 1. Ingest entries
> > >>>>>>>> 2. Ingest data size
> > >>>>>>>> 3. Scan entries
> > >>>>>>>> 4. Scan data size
> > >>>>>>>>
> > >>>>>>>> Per tablet plots
> > >>>>>>>> ----------------------
> > >>>>>>>>
> > >>>>>>>> Same as in the table plots we can display information
regarding
> > >>>>>>>>
> > >>>>>>> tablet
> > >>>
> > >>>>  servers in the tablet server page. The plots will display the
same
> > >>>>>>>> information as table plots considering data per
tablet server.
> > >>>>>>>>
> > >>>>>>>> Trace Visualization: Accumulo 1198
> > >>>>>>>> ----------------------------
> > >>>>>>>>
> > >>>>>>>> Since we are displaying graphs about each tablet
and each table
> we
> > >>>>>>>>
> > >>>>>>> can
> > >>>
> > >>>>  add
> > >>>>>>>> major and minor compaction graph to each table
and each tablet.
> > >>>>>>>>
> > >>>>>>>> Or other option is to display this in a single
graph in overview
> > >>>>>>>>
> > >>>>>>> page
> > >>>
> > >>>>  with
> > >>>>>>>> different graph lines for different tables and
tablets.
> > >>>>>>>>
> > >>>>>>>> Server type information : Accumulo 807
> > >>>>>>>> ------------------------------******---
> > >>>>>>>>
> > >>>>>>>> For displaying this informations we can add a new
page and
> display
> > >>>>>>>>
> > >>>>>>> the
> > >>>
> > >>>>  information as a table. The table should specify the network
> address
> > >>>>>>>>
> > >>>>>>> of
> > >>>>
> > >>>>>  the
> > >>>>>>>> server, server type, weather it is active or in-active
etc.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Supun...
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message