forrest-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicola Ken Barozzi <>
Subject Re: [RT] plugin infrastructure
Date Thu, 04 Nov 2004 13:30:04 GMT

It's taking me a lot of time to write this, as problems keep cropping 
up. I thought I'd send it anyway so that discussion can continue, and we 
can help each other :-)

Ross Gardler wrote:
> Nicola Ken Barozzi wrote:
>> We have now a basic plugin system, but as Ross has shown, it's not 
>> enough in practice. We have to define the points where the plugins can 
>> hook into the existing system and extend it.
>> Before going wild into defining possible scenarios, I have to 
>> understand better the reason why the current system is not enough and 
>> where we will need to extend the system.
>> Do we need plugin dependencies?
> To date I have not found a compelling reason to allow such things. So 
> far I have managed to do everything I need to with the values available 
> from core (i.e. {forrest:*} and {project:*}). My guts tell me that it 
> will be a necessity one day - but if we can avoid it all the better.

I'm thinking of the DocBook plugin and plugins that output the DocBook
as Sean has shown... so there is a dependency

   DocBook_outputting_plugin-> Docbook_plugin -> Forrest

A solution to this would be Cocoon Blocks, but they would still
need some sort of automatic resolution for us to be of use.

  Thinking out loud

Let's take for example a 'ken' source plugin and file


If I ask for


the system would give a change to match to all plugins in turn, and not
matching anything, it would try and ask for:


It asks for it, and the 'ken' plugin, finding the test.ken file would
match, thus giving us that file in DocumentDTD format.

But what if the format it gives is not DocumentDTD but DocBook? It
should match again hoping a plugin can convert it, but it's too late...

What would be optimum ATM is to have the DocBook plugin expose a
transformation service that the 'ken' plugin can use to output the
correct DocumentDTD format.

   <generate type="kengen" src="{1}"/>
   <transform src="kenxml2docbook.xsl"/>
   <transform type="docbook-plugin"/>   <----- We need this

The problem is that AFAIK Cocoon does not have this presently, they are
called virtual pipelines, and will be the heard of the Cocoon Blocks. Or
it does, but it's called resources, and they don't work between mounts IIRC.


Ok, so the thing has to be reversed: the 'ken' plugin has to ask the
DocBook plugin to generate the DocumentDTD format, but calling the
output of the ken plugin.

We can say that if a request has a 'use-plugin' parameter, that plugin
is used.

Something like this.

In the 'ken' plugin:

<!-- give it's output to whoever asks for it -->
  <match type="forrest-specific-plugin" pattern="ken-plugin">
   <generate type="kengen" src="{1}"/>
   <transform src="kenxml2docbook.xsl"/>

  <match when the ken source is found>
   <!-- This asks the docbook plugin to process the file -->

In the 'docbook' plugin:

  <!-- Here it catches the request from other plugins -->
  <match type="forrest-specific-plugin" pattern="docbook-plugin">
   <generate src="{1}{forrest-callback-target}"/>
   <transform src="kenxml2docbook.xsl"/>
   <transform type="docbook-plugin"/>   <----- We need this

Note that the details are out, it's more a TOL stuff (thinking out loud :-)

Another possibility is to have an extra system that resolves the
transformation paths, based on plugin descriptors, and generates the
code in the sitempa.xmap of the plugins.

If we don't only do this but actually *aggregate* the plugin files, we 
can use <resources> to have one plugin call the other.

   <generate type="kengen" src="{1}"/>
   <transform src="kenxml2docbook.xsl"/>
   <map:call resource="docbook-plugin"/>

To aggregate them we can use the tool that Cocoon uses.

Still dunno, these are possibilities, have to ponder more about them.

>> How are they resolved?
> Assuming that we will need them then I would propose using a similar 
> extension mechanism to that used in Eclipse. Each plugin can define 
> "extension points" which allow values to be retrieved or functionality 
> hooks to be added, I have not fully considered how to do this...
>> Where do we need to plug in?
> To date I have identified the need for three types of plugin:
> - infrastructure format (i.e. generation of tabs.xml and site.xml from 
> IMSManifest.xml or the auto site-building features in Issue tracker)
> - source format (i.e. generation of internal XML format from external 
> format, e.g., docbook etc.)
> - output format (i.e. generation of slides from our intermediate format)
> Thorsten's recent RT about Skin plugins may have identified a fourth 
> type of plugin, part of this RT is covered by the output plugins above, 
> but other parts are not.
> Other use cases may identify more plugin types.
>> Do we have to enhance existing transformations or replace them?
> No problems to date it is simply a case of finding the right mount point 
> for each type of plugin. I believe I have now found this points for the 
> above plugin types, see of 
> course, more testing with a wider range of plugins is required.

A problem may arise when a plugin wants to add metadata by examining the
source and adding to the metadata that is added by Forrest.

Maybe adding the possibility for a plugin to filter the aggregated
document should suffice, but I don't know hoe to do it.
I'd leave this use case out for now.

>> Why is the current system not working?
> It is, but with hack solutions, see FOR-344 - linked above, and my 
> proposal below this explanation.

Ok, cool, let's read :-)

> The problem with the original implementation was that the plugin sitemap 
> was being mounted before any matches were made by Forrest, as a result 
> if a plugin wanted to match all **.xml files it prevented forrest 
> specialist matchers for things like faq.xml or site.xml.

faq should start using the content aware pipelines, and this is solved.

For other special *internal* parts we should probably not use name for
matching but maybe a parameter.


> The solution was to move the plugin sitemap mount so that it came after 
> these specialist matcher but before Forrests default **.xml matcher. 
> This worked fine for the source format plugins, but prevented the 
> IMSManifest plugin from working since it needs to match the site.xml and 
> tabs.xml requests. This plugin is an example of an "infrastructure" plugin.

The above solution would solve this. Well, you could have prevented this
probably using a regexp matcher, but it's a hack :-)

> The solution to this was to mount the IMSManifest plugins sitemap before 
> the forrest matchers for site.xml and tabs.xml. As a result the 
> IMSManifest plugin has to copy the replaced matchers from Forrest (i.e. 
> site.xml and tabs.xml behaviour). However, this latter requirement can 
> be removed once we remove the permanent mount of this plugins sitemap 
> since the plugin system no builds the plugin mounts dynamically on each 
> run, that is if a site indicates it does not use the IMSManifest plugin, 
>  it will not be mounted and therefore will not override the default 
> Forrest behaviour.
> The next plugin I implemented was the s5 slideshow plugin. This 
> generates a new output format (XHTML slide presentation). This cannot be 
> mounted in the same location as the source formats since those plugins 
> are mounted in forrest.xmap, which, in turn, is mounted on a **.xml 
> match. Since output formats will be **.html, **.pdf, **.pod etc. they 
> need to be mounted from sitemap.xmap, before any Forrest specific matches.

Ok, this should be the originally intended mount point.

> My Proposal
> ===========
> Each plugin should be able to provide three sitemaps rather than the 
> current 1.
> - sitemap.xmap - this provides the infrastructure matchers (i.e. 
> site.xml, faq.xml, issues.xml), as such it will be mounted before *any* 
> of the Forrest matches. This sitemap can override any behaviour within 
> Forrest)
> - source.xmap - this provides the source matchers (i.e. **.xml), it is 
> mounted in forrest.xmap before the default forrest **.xml behaviour and 
> therefore can override that default behaviour but it will not interfere 
> with any internal Forrest infrastructure matches, or any other plugins 
> infrastructure matches.
> - output.xmap - this provides (surprise!) the output matchers (i.e. 
> **.html, **.pdf, **.slides), it is mounted before any of the default 
> matchers for Forrest and so can override this default behaviour
> Ideally, each plugin should only provide one of these three sitemaps. I 
> believe this will help us to define a sensible granularity for the 
> plugins. However, further exploration of use cases may show otherwise 
> and therefore we should make it legal (but not encouraged) to implement 
> more than one of them.

I tend to like the concept, even if it may not result to be technically
needed. Just thinking out loud.

> Avoiding Plugin Conflicts
> -------------------------
> Clashes between plugins can occur. For example, the simplified-docbook 
> and full docbook plugins may try and process the same files. In this 
> instance the one that is mounted first will take precedence. Plugins are 
> mounted in the order they appear in the project.required.plugins 
> property, therefore the mounting order and therefore processing 
> precedence is under user control.


Easy, simple, effective.

An alternative would be that plugins define conversion services and that 
there is a resolution system that pipes one in the other... please stop 
me! :-D

> Remaining problems
> ------------------
> The mount point of the "infrastructure" plugins means that plugins of 
> this type can prevent Forrest from behaving correctly. These types of 
> plugins should be considered "advanced" and authors must be very careful 
> when building them - note the same is true for the existing mount point 
> of the project sitemap - perhaps we should consider using the same 
> design for project extensions as we use for plugin extensions (a whole 
> new RT I think)

Let's leave this out fo rnow.

> - Plugin resources are not 
> copied to site - in order to fix this problem I believe each plugin 
> should implement a site-build.xml file that provides a target for 
> copying any required resources into the built site. This is not 
> necessary when running dynamically.

Well, this is another RT altogether, about how to generate a site with 
non-linked resources.

Nicola Ken Barozzi         
             - verba volant, scripta manent -
    (discussions get forgotten, just code remains)

View raw message