avalon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leif Mortenson <l...@silveregg.co.jp>
Subject Re: [Fwd: Re: New AltProfile package]
Date Sat, 09 Mar 2002 04:20:03 GMT


Berin Loritsch wrote:

> You have converted me.


:-)


>  >> I don't know all of the pros and cons of the way Leif implemented it,
>  >> but the fact that we have something that works is really cool.
>  >
>  >
>  > Thanks.  I am sure it is far from perfect still.  But hopefiully we can
>  > get it in that
>  > direction soon :-)
> 
> If you are half the perfectionist I am, we will never be satisfied ;P
> 
> Eventually, we do have to release something.  As long as we can get the
> API where we want it, we can fine tune the implementations as we go on.


Sounds good.  I'm not smart enough to get to perfect anyway without other
people using it and making lots of requests and suggestions along the
way :-)

>  >> We don't want to profile all the time--but we do want to be able to 
> turn
>  >> profiling off an on multiple times during the life of a system.  As it
>  >> stands now, it is only useful for standalone programs.
>  >
>  >
> 
> <snip comment="Profiling can be turned off and on at will."/>


Gotcha.  Everyone seems to agree there.  I need to take a loo kat the core
again figure out to make this really smooth.

It shouldn't be all or nothing.  Its more like certain ProfilePoints need
to be turned on and off at will.  Or at least sets of them.

>  > One thing that it does not do a good job of right now is adding and
>  > removing
>  > ProfilerSamples at run time which reside inside of the ProfilerManager.
>  > Outside
>  > is no problem.
> 
> Does the ProfilerManager use itself to manage those profile samples, or
> does it short-circuit things to make it "easier" on itself?


All sampling is done inside the ProfileSamples.  I use listeners to keep
the flow of execution for new data points flowing through as small a number
of objects as possible.  Makes it more efficient.

The problem is that if you have pieces of your management system which are
relying on say the sample data for free memory.  Then you need a way to lock
that ProfileSample so it will never stop collecting data and it will never
be removed.

Then there is also the case where you have several client interfaces
connecting and registering SampleProfiles.  These samples are required by
those clients so that they can each display the data requested.  In some
cases, the data should be transcient. Ie. It starts capturing when
requested and stops when released.  Another case though would be to be
able to let the client register a semi-permanent profilesample so they
could disconnect their client, then come back later and see the data at
any time.  It would be nice if such data also survived JVM restarts like
the current ProfileSamples.

Then Marcus would also like to be able to have scheduled Profile Samples
which would collect data for certain periods of time each day. (?)

To make matters worse, the interface to do all this has to be fast, simple
and ellagent. :-)


>  > The question is how should this API be secured.  Who should have access
>  > to it
>  > etc.  Currently, which ProfileSamples are to be made available is up to
>  > the system
>  > administrator.  If anyone can add or remove these ProfileSamples at
>  > will, it could
>  > break other systems which are using them.  Anything added outside of the
>  > ProfilerManager is controlled by the user code which does the adding.
> 
> Could you explain your concerns a little better?  The only three classes
> of people who should have the right to view profiling data are the
> sys-admin, the tester, and the developer.  Any sampling point that
> caters to a different class of people is not a good sampling point.


I was thinking that there are lots of places where the profile data can
be used for business purposes etc.  Say on a e-commerce site.  You may want
to collect data which showed things like the number of times an item was
placed into a visitor's basket or purchased.  Page views per second etc.

Data sets like this become a resource of the system and should not be able
to be suspended or removed.

> We can use JAAS to ensure that the GUI can be attached by a properly
> approved user.


Not familliar with JAAS yet, but it should be possible to allow different
actions on the profile data depending on user role.


>  > While some profile data is used for "profiling".  Other profile data is
>  > needed as a
>  > resource for an application.  For example a system administration
>  > application
>  > may have an HTTP interface which contains servlets showing the system
>  > memory
>  > usage over time as well as the access rates.  This is information which
>  > is being used
>  > as a resource and must always be collected for the system to behave
>  > correctly.
> 
> We would have a different type of listener for that.  Typically,
> anything that creates charts to serve over the web publish their results
> every so often.  It is atypical if the results are published more often
> than every half hour.  It is more typical if the results are published
> every six hours or even daily.


Actually clients that need data at a vixed interval like that should really
not be registering as listeners.  Rather they should be obtaining a
reference to the appropriate ProfileSampleDescriptor and then requesting
a snapshot at the appropriate time.


> That means we need listeners that can be instantiated by the servlet
> container (assuming we provide an HTTP server based on asynchronous
> sockets....), and those listeners publish their results to a specific
> directory in the filesystem.
> 
> Also, that means we have some listeners like the GUI that can be
> attached at will to a Servlet that is built around the profiling code
> to determine it's specific health.


I still need to work out some issues for how to deal with listeners for
remote systems.  Right now if a remote listener registers itself with a
profile point, the code which uses the profile point would be really
suffer a big performance hit.  So I don't think that should be allowed.
Even the current ProfileSample Listeners could have a lot of traffic.
I think that anything that goes across RMI should be done asyncronously
to prevent a performance hit in the code being profiled.  Currently, the
same thread flows up through the ProfilePoint to all the listeners and
ProfileSamples and back.  This is quite efficient now, but it would not
be if there was a two way socket communicate thrown in there.


One idea is to allow clients to connect as a listener with a requested
update interval.  Say 1 second.  As well as a flag which triggers a call
each interval, or only for intervals which have new data.  The
ProfilerManager would then use a pool of runner threads to service those
listeners in the background.

I think those kinds of events would actually serve all of the requests
that I have heard so far.  Actually a little better than the current
Listeners which actually provide too much information.

Currently the charts in the ProfilerFrame have their own threads which
request snapshots each second.  If I modified them to register as
themselves as Interval Listeners, as above, then the timing of their
updates would be driven by the ProfilerManager, removing the need for
them to each have a runner thread.

Thoughts?


>  >> We need a way to
>  >> make it useful for situations where we can't directly control what
>  >> classloader is used to load the system (i.e. Servlet environment, etc.)
>  >
>  >
>  > We are currently using it in a Servlet system running under Jo.  We had
>  > a couple
>  > problems getting the XML serialization stuff working because of
>  > conflicts with
>  > the jars that came with Jo, but other than that it works great.  Those
>  > problems
>  > are only coming up because of the ability to save the state of the
>  > samples in the
>  > ProfilerManager and saving of the desktop of the ProfilerFrame.  The
>  > ProfilerFrame will soon be broken into a different program so the second
>  > should
>  > go away.
> 
> :)
> 
> That is not an uncommon problem.  People who use Cocoon are well
> familiar with those issues.


Yes, but I'm sure that doesn't stop them from grumbling about them any
though :-)


I was actually trying to think of how to set up the Profiler in something
like Phoenix.  Currently, you could have one management console.  But each
app would have to have its own ProfilerManager.   I don't think there is
a way to create a single ProfilerManager which would be shared by multiple
application because they were each loaded by a different Classloader.

Still don't have any ideas for how to be able to profile a Phoenix system
which consists of several applications....

>  >> It's being tied to the config file that I am not a fan of.  If the
>  >> config file is merely a way to set up the last suite of ProfilePoints
>  >> we were monitoring, then excellent.  I just don't want to be tied to
>  >> what the config says I have, if my software is changing.
>  >
>  >
>  > Was this made clearer above?  The profilable attribute on each component
>  > is required to allow an administrator to decide whether or not a 
> component
>  > should be visible to the ProfilerManager and its APIs.  It also makes it
>  > possible
>  > to configure the name which will be used to reference the component 
> in the
>  > profiler space.
> 
> Ok.  However, the more I think about it, we should allow the
> profilemanager make those decisions.  Many systems don't support auto
> reloading of configuration files while they are still running.  It is
> unfortunate to have to restart a system just to expose more profile
> points.


I agree, but currently, I need to have a way of assigning names to the
different Profilables.  There is not any reason why all Profilable
components shouldn't have a profilable attribute set.  It just enables
their registraton with the ProfilerManager at initialization.  That can't
really be done after the fact anyway.

Profilables only take a performance hit when listeners are added to their
ProfilePoints.

If there was a reliable way for the ProfilerManager to come up with unique
names for all Profilable components, then we could get rid of the attribute
and just have them always be registered.

Any ideas on how to generate a name which would work with components inside
of selectors etc?  I was thinking about just using the names of the elements
in the configuration file for the components, but that is not so clear when
the roles file is not being used.  Also components can be added to ECM without
using the config file right?

> What should be configurable is the ProfileManager being used.  For
> instance, the DefaultProfileManager is great for development use, but
> we should be able to plug in a SecureProfileManager that will require
> listeners to authenticate themselves before being attached.  Even then,
> results should be sent encrypted if they are sent over the wire.


Ok, but I don't think this is a configuration issue.  It is rather left up to
the programmer to decide which implementation of the ProfilerManager
interface should be passed to the PCM in the setProfilerManager method.
It is a little too low level to be changed at runtime.

> That way, we can enable remote monitoring, while aleviating security
> administrator's fears of publicly exposing critical information.  (that
> can kill a project--I've had it happen to me).


I think that role support should be built into the PM, but the encryption issue
can probably be handled by the code which actually services remote requests.
Maybe register an AltRMIClient in the same JVM to the PM.  Then remote clients
would connect to that AltRMIClient via RMI.  You could do the same for HTTP or
HTTPs clients for example.  That makes it easy for new protocols to be added
without adding any clutter to the PM.  Each connector would just be connecting
to the PM using the normal APIs

>  > I agree that there should be a way to serialize output data in whatever
>  > format you
>  > want.   But do not confuse that with the profiler.sampledata file which
>  > is currently
>  > being saved at regular intervals and shutdown by the ProfilerManager.
>  > This file
>  > really only has one purpose.  Making it possible to preserve the 
> state of
>  > ProfileSample data across invocations of the JVM.  That means that the
>  > file must
>  > be read as well as saved.  If this file format can be changed then the
>  > reading also
>  > becomes very complicated.  I am all in favor of adding a way to make 
> other
>  > snapshots to any file format.  Maybe implement something like the
>  > targets in the
>  > logger.
> 
> So, in order to solve this problem, we should have additional listeners
> who's responsibility is to save the data in whatever format we need?


Ok, it would be nice if this could piggy back the regular client interfaces
though.

Maybe have a report generator client which requested to be notified every 30
minutes by a certain ProfileSample...  That report could then write out a CSV
file or something.

Cheers,
Leif


--
To unsubscribe, e-mail:   <mailto:avalon-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:avalon-dev-help@jakarta.apache.org>


Mime
View raw message