gump-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam R. B. Jack" <aj...@apache.org>
Subject Re: Gump3 ideas/questions
Date Fri, 20 May 2005 20:33:59 GMT

> > I think in these (early days, maybe through-out the life of Gump3) the
> > toughest part is the "property messaging protocol(s)". Basically Gump3
> > internals are based upon trust & respect between plugins -- like a good
OSS
> > project :-) -- but it doesn't (yet) have the internal equivalents of
> > communications forums of wikis/blogs/docs, nor the history of commit
mails,
> > mail archives, nor SCMs.
>
> I love that description. Nope, we don't have that. But we can reuse all
> those tools just by localizing the trust and respect into an explicit
> file! We need

Funny how I wrote it w/o quite making the leap of using them exactly, I was
expecting analogous components in Gump3. Could be interesting.

>  gump/model/propertymessaging.py
>
> "This module contains *all* the getters/setters that plugins use for
> adding data to the model or getting it from the model."

Yup, I like it. I think we'll need a few such "protocols" -- i.e. the
build/status as one protocol, with "artifacts" (classpaths, reposities,
generating) perhaps another. I'm sure there will be many.

> We get SCM/commit mails you get by the file being in SVN. We also get
> versioning of the protocol that way: just tag revisions of the file:
>
>   svn cp
>
https://svn.apache.org/repos/asf/gump/branches/Gump3/pygump/gump/model/propertymessaging.py
>
https://svn.apache.org/repos/asf/gump/tags/Gump3-propertymessage-protocol-3.0
>

Yes, interesting.

That said, I was actually refering to plugins sharing the model (as
developers share the code, via SCM) and how one plugin's efforts, setting
properties, might (in plugin's real time) communicate. Just as we trust each
other to move things towards a common goal, and respect each others changes
(even reversions), and communicate via lists/blogs/SCMs -- I wonder if
plugins could. Perhaps some named dictionaries on model objects, that allow
plugins to say (I created  x/y/z -- and modified a/b/c). I could see the
"core" (algorythm/walker) being able to do some nifty/helpful
tracing/debugging with that. If Gump3 ever gets so "big" (flexible) that it
loads plugins on the fly, from developers whove never spoken, I could see
this being very useful. For now, it is just a wild fancy. :-)

> I've actually been changing that a little (make sure you svn upped
> everything :-D). From memory,

Yup, always do. I (eventually) stumbled on the in code docs for your changes
also, so got an understanding. Thanks.

> Plugins should make an attempt to handle an expected failure, ie they
> should try and continue processing, and set the "failed" property on the
> model element that failed. They should raise exceptions on unexpected
> failure. They don't do anything else.
>
> The Builder nor the Walker nor any of the "core" code have anything to
> do with this, *except* the algorithm (in algorithm.py). All decisions
> about what should happen on a failure, all intelligence on what a
> failure *means* or what *consequences* a failure should have (for
> example, if you can't check out a module no use trying to build it) are
> made by the algorithm.

Ok, then time for me to fess up. One thing I did was (in BuilderPlugin) was
setting the status to failed (using your protocol setter) if an exception
was thrown. I wondered (and added a #TODO) if we needed a
BuilderErrorHandler for that instead. I now see you have a third
alternative, and maybe we take the exception catch of out BuilderPlugin all
together.

> I'll be frank: I don't think so. I think that if properties become too
> big, they should be turned into functionality to access properties. For
> example, instead of keeping logs in the model, you store them on disk or
> in a database, and you load them into the plugin space (instead of onto
> the model) as you need them:

This is exacly the example I think of where we might want to use the
property(get,set,del,'doc string') capability. I know I'm a little enamoured
w/ this beastie, but as we use properties more and more, I could see it be
interesting for a plugin to annotate a model object with both data and the
getter/setter/delete/docs logic behind it (e.g. to read from file on
demand). Other users don't need to know the implementation details, just
that it provides/consumes a string.

> Hmm. While typing that, I realized that you can actually build a reaper
> that way, and its just a plugin that looks for the "big" chunks of
> memory, writes them to your object db of choice (ie python's "shelf"
> module or whatever), then adds an accessor to load them back into memory.

I think we will want a Reaper, not so much 'cos of the size of properties
(as discussed above) but the shear volume of them. Gump2 slowed to mollasses
as it grew, before I spent a month or so fighting it, and being careful to
assit GC. Python's runtime gets clogged with a few hundred thousand objects,
and (especially when Gump2 creates DOM trees for documentation during the
build) we were easily hitting that. Python GC is quite the "magic" I'd wish
it to be.

> However, one wonders if this is really more efficient than having the OS
> do the swapping out of memory onto disk. I'm guessing that the OS maybe
> has a hard time figuring out what it can reclaim from python (ie what is
> the good stuff to swap out).

Yup, it is Python GC that really needs the assist (at least in Python2.3).

> One thing Gump3 doesn't do is keep references to objects anywhere in the
> plugins, the utility code, or the algorithm. That means its GC should be
> a lot more predictable. Precisely by storing everything in the model and
> consistently using a (stateless-)visitor pattern I think we will have to
> worry a lot less about objects marked as "used" when they really aren't.

I think Gump3's direct use of properties (not adding extra helper classes w/
methods w/ wrapper/complex objects, etc. etc.) ought help the most.

> > Anyway, how might a Reaper fit in?
>
> Just another plugin :-)

I was thinking of the "update & build & document[to DB]" chain (say) for
ant. Once done, are all the details of the build, that documentation used,
still needed? I meant reaping to death, not paging out. We'll see if it is
needed, but somehow the algorythm would need to know when a model is "used
for this run", and/or the Reaper plugin would and/or we'd need a special
plugin the algoruthm "understood".

> Did I mention I hate cygwin? :)

Me too (now). Top of my wishlist (since I don't have the spare dosh to fork
out for a box to install Linux on) is to get off it. Being Python on
Microsoft is a pain (w/o some posix abilities) but it is better (if we can
get there) than being on  Python on Cygwin on Microsoft, it is too
confusing. Only real victim would be your bash "gump" script...

> Pfew. Really need to get back to work...

Thanks for taking the break. :-)

regards

Adam


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


Mime
View raw message