accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Supun Kamburugamuva <supu...@gmail.com>
Subject Re: GSOC: Monitor Improvements
Date Mon, 22 Apr 2013 15:04:48 GMT
That sounds interesting. To clarify the requirement, we can have a process
that exposes the same JMX mbeans as the the real server and monitor can
plug in to this process.

Thanks,
Supun..


On Mon, Apr 22, 2013 at 10:57 AM, Josh Elser <josh.elser@gmail.com> wrote:

> That would be pretty sweet, actually. Potentially parallel to what you
> want to do, Supun, but cool nonetheless.
>
> I could see a lot of benefit by having some process that could emulate the
> output from a non-trivially-sized Accumulo cluster on a single box.
>
>
> On 4/22/13 10:43 AM, Eric Newton wrote:
>
>> You could mock the stats collection.
>>
>> -Eric
>>
>>
>> On Mon, Apr 22, 2013 at 10:41 AM, David Medinets
>> <david.medinets@gmail.com>**wrote:
>>
>>  The average developer probably can't access a large cluster with hundred
>>> of
>>> nodes. Is there a way to simulate this?
>>>
>>>
>>> On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <eric.newton@gmail.com>
>>> wrote:
>>>
>>>  Another thing to consider is scale.  On large clusters (many hundreds of
>>>> nodes), more data is not helpful for visualization.  Instead, summaries,
>>>> averages and outliers are important.
>>>>
>>>> For example, if one node is consistently slow, it is better to know that
>>>> than to see one graph with low numbers in a sea of graphs.
>>>>
>>>> If the monitor collects information using JMX, collection time for each
>>>> node would be a good thing to know, too.
>>>>
>>>> -Eric
>>>>
>>>>
>>>> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <josh.elser@gmail.com>
>>>>
>>> wrote:
>>>
>>>> Supun,
>>>>>
>>>>> Yup, very much so. Having a way to consume any and all metrics via JMX
>>>>> would simplify things for any consumers (internal or external).
>>>>>
>>>>>
>>>>>
>>>>> On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
>>>>>
>>>>>  Hi Josh,
>>>>>>
>>>>>> Thanks for the suggestions. I'll incorporate these to the proposal.
>>>>>>
>>>>>> Another area I would like to work is on JMX. There is a Jira that
says
>>>>>>
>>>>> to
>>>>
>>>>> replace the Monitor calls from Thrift to JMX (Accumulo 694). Do you
>>>>>>
>>>>> think
>>>>
>>>>> this is a good addition to the Monitor?
>>>>>>
>>>>>> Thanks,
>>>>>> Supun..
>>>>>>
>>>>>>
>>>>>> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <josh.elser@gmail.com>
>>>>>>
>>>>> wrote:
>>>>
>>>>>   Supun,
>>>>>>
>>>>>>> Looks good! Can I make some suggestions/comments?
>>>>>>>
>>>>>>> For: "Per table plots: ACCUMULO-594", I'd also like to see minor
>>>>>>> compactions, major compactions, index cache hit rate, and data
cache
>>>>>>>
>>>>>> hit
>>>>
>>>>> rate per table (same graphs that are displayed system-wide when you
>>>>>>>
>>>>>> visit
>>>>
>>>>> http://${MONITOR_HOST}:50095/.
>>>>>>>
>>>>>>> For "Per tablet [server] plots", it would be neat if you could
also
>>>>>>> extract some general statistics like top N least performing,
top N
>>>>>>> highest
>>>>>>> performing, etc. tablet servers. Ideally, this could correlate
with
>>>>>>> servers
>>>>>>> that may be having problems :).
>>>>>>>
>>>>>>> Do you see these proposed changes as being sufficient for 3-4
months
>>>>>>>
>>>>>> of
>>>
>>>>  40hrs/week work? If you plan to really dig into these changes
>>>>>>>
>>>>>> (perhaps
>>>
>>>>  reworking components of the monitor itself), I could perhaps see
>>>>>>>
>>>>>> this.
>>>
>>>> Do
>>>>
>>>>> you have any ideas for more lofty goals that you could pursue as
>>>>>>>
>>>>>> well?
>>>
>>>> I
>>>>
>>>>> don't want you/us to get one month into things and see you complete
>>>>>>> everything we initially planned to accomplish :)
>>>>>>>
>>>>>>> - Josh
>>>>>>>
>>>>>>>
>>>>>>> On 04/21/2013 10:37 AM, Supun Kamburugamuva wrote:
>>>>>>>
>>>>>>>   Hi all,
>>>>>>>
>>>>>>>> I would like to start writing the proposal for the GSoc.
I've put
>>>>>>>> together
>>>>>>>> some initial high level goals of the project. Please let
me know
>>>>>>>>
>>>>>>> what
>>>
>>>> I
>>>>
>>>>>  can
>>>>>>>> improve.
>>>>>>>>
>>>>>>>> Per table plots: Accumulo 594
>>>>>>>> ---------------------
>>>>>>>>
>>>>>>>> The goal of this is to display plots that explains the various
>>>>>>>> activtities
>>>>>>>> that happens per table. When we go to the tables page of
the monitor
>>>>>>>>
>>>>>>> and
>>>>
>>>>>  go
>>>>>>>> to a specific table it displays some information in a table
format.
>>>>>>>>
>>>>>>> We
>>>
>>>>  can
>>>>>>>> argument this information by showing graphs for
>>>>>>>>
>>>>>>>> 1. Ingest entries
>>>>>>>> 2. Ingest data size
>>>>>>>> 3. Scan entries
>>>>>>>> 4. Scan data size
>>>>>>>>
>>>>>>>> Per tablet plots
>>>>>>>> ----------------------
>>>>>>>>
>>>>>>>> Same as in the table plots we can display information regarding
>>>>>>>>
>>>>>>> tablet
>>>
>>>>  servers in the tablet server page. The plots will display the same
>>>>>>>> information as table plots considering data per tablet server.
>>>>>>>>
>>>>>>>> Trace Visualization: Accumulo 1198
>>>>>>>> ----------------------------
>>>>>>>>
>>>>>>>> Since we are displaying graphs about each tablet and each
table we
>>>>>>>>
>>>>>>> can
>>>
>>>>  add
>>>>>>>> major and minor compaction graph to each table and each tablet.
>>>>>>>>
>>>>>>>> Or other option is to display this in a single graph in overview
>>>>>>>>
>>>>>>> page
>>>
>>>>  with
>>>>>>>> different graph lines for different tables and tablets.
>>>>>>>>
>>>>>>>> Server type information : Accumulo 807
>>>>>>>> ------------------------------******---
>>>>>>>>
>>>>>>>> For displaying this informations we can add a new page and
display
>>>>>>>>
>>>>>>> the
>>>
>>>>  information as a table. The table should specify the network address
>>>>>>>>
>>>>>>> of
>>>>
>>>>>  the
>>>>>>>> server, server type, weather it is active or in-active etc.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Supun...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>


-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message