cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Mazzocchi <>
Subject Re: [RT] Escaping Sitemap Hell
Date Fri, 07 Jan 2005 05:49:56 GMT
Daniel Fagerstrom wrote:
>  (was: Splitting xconf files step 2: the sitemap)
> Although the Cocoon sitemap is a really cool innovation it is not 
> entierly without problems:
> * Sitemaps for large webapps easy becomes a mess
> * It sucks as a map describing the site [1]
> * It doesn't give that much support for "cool URLs" [2]
> In this RT I will try to analyze the situation especially with respect 
> to URL space design and then move on to discuss a possible solution.
> Before you entusiasticly dive into the text:
> * It is a long RT, (as my RTs usually are)
> * It might contain provoking and hopefully even thought provoking ideas
> * No, I will not require that everything in it should be part of 2.2
> * No, I don't propose that we should scrap the current sitemap, actually 
> I believe that we should support it for the next few millenia ;)

See my comments intermixed.

>                            --- o0o ---
> Peter and I had some discussion:
> Peter Hunsberger wrote:
>> On Tue, 04 Jan 2005 13:25:05 +0100, Daniel Fagerstrom 
>> <> wrote: 
> <snip/>
>>> Anyway, sometimes when I need to refactor or add functionallity to 
>>> some of our Cocoon applications, where I or colleagues of mine have 
>>> written endless sitemaps, I have felt that it would have been nice if 
>>> the sitemap would have been more declarative so that I could have 
>>> asked it basic things like geting a list of what URLs or URL pattern 
>>> it handles. Also if I have an URL in a large webapp and it doesn't 
>>> work as expected it can require quite some work to trace through all 
>>> involved sitemaps to see what rule that actually is used. 
>>> Of course I understand that if I used a set of efficient conventions 
>>> about how to structure my URL space and my sitemaps the problem would 
>>> be much less. Problem is that I don't have found such a set of 
>>> conventions yet. Of course I'm following some kind of principles, but 
>>> I don't have anything that I'm completely happy with yet. Anyone 
>>> having good design patterns for URL space structuring and sitemap 
>>> structuring, that you want to share? 
>> We have conventions that use sort of type extensions on the names: 
>>, patient.list, patient.edit where the search, list,
>> edit (and other) screen patterns are common across many different
>> metadata sources (in this case patient). We don't do match *.edit
>> directly in the sitemap (any more) but I find that if you've got to
>> handle orthoganal concerns then x.y.z naming patterns can sometimes
>> help.
> Ok, lets look at this in a more abstracted setting:
> Resource Aspects
> ================
> In the example above we have an object or better a _resource_, the 
> patient that everything else is about. The resource should be 
> identifyable in an unique way in this case with e.g. the social security 
> number.

First big mistake: you think that http-based URIs and http-based URLs 
are the same thing.

Well, WRONG.

There is nothing that says that every http-URI should be automatically 
treated as a URL. This is a very commmon misconception, but nevertheless 
a big one.

> There are a number of _operations_ that can be performed at the patient 
> resource: show, edit, list, search etc, (although the search might be on 
> the set of patient rather than a single one).
> The resource has a _type_, patient, that might affect how we choose to 
> show it etc.

Secong mistake: it is a architectural design issue to *avoid* adding a 
type to a URI. These are three separate issues:

  1) how to resolve a URI into a URL
  2) how to negotiate the content of that URL
  3) how to map that returned URL metadata (the HTTP response headers) 
to a recognized type or format.

combining them into one is just a really poor way to use the web 

> There are in general other aspects that will stear how we render the 
> response when someone asks for the resouce:
> * The _format_ of the response: html, pdf, svg, gif etc.
> * The _status_ of the resource: old, draft, new etc.
> * The _access_ rights of the response: public, team, member etc.
> There are plenty of other possible aspect areas as well.

> Cool Webapp URLs
> ================
> I searched the web to gain some insights in URL space design. It soon 
> become clear that I should re-read Tim Berners-Lee's clasic, "Cool URIs 
> don't change" [2]. I must say I wasn't prepared to the chock, I had 
> completely missed how radical the message in it was when I read it the 
> last time. 
> I can also recomend reading [3], a W3C note that codifies the 
> message from [2] and some other good URI practices into a set of 
> guidelines.

I suggest you to read

> So what is an URI? According to [3]:
>  A URI is, actually, a //reference to a resource, with fixed and 
> independent semantics/ /.
> This means that the URI should reference to a specific product, 
> _always_. 


How to get a reference out of an itentifier is a totally different thing.

> Independent semantics means that a social security number is 
> not enough, it should say that it is a person (from USA) as well. See 
> [3] for the philosophical details.

Pfff, independent semantics doesn't mean anything. A perfectly valid URI is


> * The URI should be easy to type

What the hell does this mean?

is easier to type than

but which one is "better"? They both locate the same resource, but which 
one of them identifies it better?

> * It should not contain to much meaning, especially not about 
> implementation details
> Now I try to apply the ideas from [2] and [3] on the different resource 
> aspects mentioned above. When I use words like "should" or "should not" 
> without any motivation it means that I believed in the motivation from 
> the gurus in the references ;) I will try to motivate my own ideas ;)
> What I'm going to suggest might be quite far from how you design your 
> URL spaces. It is certainly far from the implementation detail plauged 
> mess that I have created in my own applications.
> The Resource
> ------------
> The idea is that an URL identifies a resource. For the patient case 
> above it could be:
> If we use a hierarchial URI space like /person/123456789, the "parent" 
> URIs e.g. /person should also refer to a resource. 

There is *NO SUCH THING* as a parent URI, because URIs do not have the 
notion of paths. It is a *convention* that it was established by early 
web server implementations (and that apache httpd perdured) that the / 
in the paths got automatically mapped to the / in the file system or in 
a hierarchical system where the / is used as a fragmentor for hierachy 

There is *NOTHING* in any web spec that says this is the rule or, for 
that matter, that this is a good thing.

/ is a "separator" in fact, from a URI point of view


show no difference in identification power.. which is what URIs do: they 

> Its in most cases not 
> a good idea to put a lot of topics classification effort in the URI 
> hierarchy. Classifications are not unique and will change according to 
> changing interests and world view.

This is true. But it is also true that, if you follow this reasoning, 
you should not be using http:// URIs at all!

In fact, what happens to a URI when say, two hospitals merge and they 
decide that it's in their best interest to get rid of the previous 
references of the names, including those in the URIs?

This is the reason why a lot of people prefer URNs over http-URIs, for 

  1) the handle system:
  2) the LSID system:
  3) the DOI system:

TimBL believes that the above systems are just a different way to skin a 
cat and they don't really solve anything (even if he agrees on the 
problem that the domain part of http-URIs is the weakest part of an 
http-URI, in terms of long-term persistence)

Also, you should take a look at 'Dynamic Delegation Discovery System' 

which aims to become the standard way to translate a URI into a URL.

> Operations
> ----------
> What about the operations on the resource: list, search, edit etc? I 
> find the object oriented style in WebDAV elegant where you use one URL 
> together with different HTTP methods to perform different operations. 

It's not the OO style of WebDAV, but it's the design of HTTP. Here is 
another example of somebody ruining a perfectly great design by not 
getting it: the browsers only allowed people to overload the actions in 
forms, but never in anchor tags and the browsers never allowed 
javascript to change that either.

> Sam Ruby also have some intersting ideas about using URLs to identify 
> "objects" and different SOAP messages for different methods on the 
> object in his "REST+SOAP" article [4]. But neither adhoc HTTP methods or 
> XML posts seem like good candidates for invoking operations on a 
> resource in a typical webapp. So maybe something like:
> /person/123456789/edit or
> /person/123456789.edit or
> /person/123456789?operation=edit
> is a good idea.
> Resource Type
> -------------
> Should the type of the resource be part of the URI? 

Absolutely not!

> We probably have to 
> contain some type info in the URL to give it "independent sematics" 
> (person e.g.). But we should not put types that might change like 
> patient, manager, project-leader etc in the URL. And we should 
> especially avoid types that only have to do with implementation details 
> like what pipeline we want to use for rendering the resource.
> Format
> ------
> Cocoon especially shines in handling all the various file name 
> extensions: .html, .wml, .pdf, .txt, .doc, .jpg, .png, .svg, etc, etc. 
> But I'm sorry, if you want cool URLs you have to kiss them godbye as 
> well ;)

This is, again, another one of those major screwups from some browsers 
(mostly IE) where the "extension" of a URL (as such a thing existed!) 
was used to identify the mime-type instead of the response headers.

> It might be a good idea to send a html page to a browser on a PC and a 
> wml page to a PDA user. But you shouldn't require your user to remember 
> different URLs for different clients, thats a task for server driven 
> content negotiation.
> Using .html is not especially future proof, should all links become 
> invalid when you decide to reimplement your site with dynamic SVG?
> Often it is good to provide the user with a nice printable version of 
> your page. But why should you advertice Adobes products in your URLs. 

Unfair: many non-adobe things produce PDF and it's a royalty-free 
specification to use.

> A 
> few years ago it was .ps or .dvi from academic sites and .doc in 
> comersial sites. Right now it happen to be .pdf but will that be forever?
> Same thing with images, the user don't care about the format as long as 
> it can be shown in the browser (content negotiation), neither should you 
> make your content links or (Googles image search) be  dependent on a 
> particular compression scheme that happen to be popular right now.
> There are of course cases where you really whant to give your user the 
> abillity to choose a specific format. Then a file name extension is a 
> good idea. If you happens to maintain 
> its ok to put some .pdf there 
> e.g. ;)
> But in most cases file name extensions is an implementation detail that 
> not is relevant for your users.

This is correct. Although a URL that might break in the future but shows 
me a page in my browser today is better than a URL that might not break 
tomorrow but doesn't show me anything at all today ;-)

> Status
> ------
> The status will by definition change, and that make your URL uncool if 
> the status was part of the URL.
> Access Rights
> -------------
> Access rigths will often change for a document. I know it is easy to 
> write path dependent rules for access rights in most webserver 
> configuration files. But you expose irrelevant implementation details 
> and its not future proof.
> Am I Really Serious?
> --------------------
> Why should a webapp URL be cool and future proof? Well, its the 
> interface to your webapp. We agree that we shouldn't change interfaces 
> in Cocoon at a whim, why should we treat the users of our webapps 
> differently? And like it or not, usefull software sometimes lives for 
> decades. If you build useful webapps you should consider planing ahead.
> Currently we are all used with webapps that uses the most horrible URLs 
> containing tons of implementation details and changing every now and 
> then. But it is not a law of nature that it must be like that. It is 
> mainly a result of webapp development still being immature and the tools 
> being far from perfect. Of course the user should be able to bookmark a 
> useful form or wizard.
> Also I believe that exposing implementation details in ones URLs is at 
> least as bad as making all member variables public in Java classes. It 
> makes your webapp monolithic and fragile.

To get this straight: I totally agree that a cool URL scheme is a great 
thing and I also think that the best URL scheme is something like

and that's it... that's the only way never to change anything because 
those numbers are the only 'semantically neutral' thing that you can do?

But still, my blog news URLs are the form of

which have several problems:

  1) we might forget to register the domain and somebody might steal it 
from us

  2) well, my name might change (but that's unlikely)

  3) the company that has a trademark on linotype might sue me

  4) I might decide to add other types of idems to my blog, like images 
or articles or whatever else... then news/id/ would seem awkward

but the best part is the number, chosen to be incremental and unique in 
that space.

>                            --- o0o ---
> You might find the views expressed above rather extreme and maybe 
> unpractical. As indicated above they are also far away from what I 
> curently do in my webapps. But I have for quite some time thought about 
> how to fight the to easily increasing entropy in the webapps we develop. 
> I have suspected that badly designed URL spaces has been part of the 
> trouble. And when I re-read Tim BLs classic I suddenly realized that the 
> habit of exposing implementation in the URLs might be at the root of the 
> evil.

There is truth in this, but what I found irritating was the lack of 
understanding of the difference between a URI and a URL.

Cocoon's internals show some of this too (and I have to admit that I 
understood what URIs really were only after starting to work on the 
semantic web) but this should not be perpetuated further.

> If this realization will survive the contact with your comments and 
> other parts of reality is of course to early to tell ;)
> Does Cocoon Support Cool URLs?
> ==============================


> But how does Cocoon support the above ideas about URL space design?
> Well, in some way one could say that it supports it. The sitemap is so 
> powerfull that you can program most usage patterns in it in some more or 
> less elegant way. But AFAICS, writing webapps following the URL space 
> design ideas above would be rather tricky. So I would say that Cocoon 
> doesn't support it that well. 

I rather strongly (and probably not surprisingly) disagree with this 

> The main reasons are:
> * The sitemap is not that usefull as a site map

How is this making it worse to support "cool URLs"?

> * The sitemap gives excelent support for choosing resource production 
> implementation based on the implementation details coded into the URL, 
> but not for avoiding it

wrong! that's why we have pluggable matchers! the fact that you choose 
to match by URL is your choice, not an architectural decision!

> * The sitemap mixes site map concerns with resource production 
> implementation details

Yes, the cocoon sitemap describes how resources get produced in the 
pipelines.... but what is the site map you are talking about? a 
collection of all the resources available on the site? or just the URL 
matchers without anything else?

> Is it a Map of the Site?
> ------------------------
> The Forrest people don't think that the sitemap is enough as map of the 
> site. They have a special linkmap [1] that gives a map over the site and 
> that is used for internal navigation and for creating menu trees. I have 
> a similar view. From the sitemap it can be hard to answer basic 
> questions like:
> * What is the URL space of the application
> * What is the structure of the URL space
> * How is the resource refered by this URL produced

Hold it right there!

If you think that understanding the URLspace of the application for a 
sitemap is hard, then what about PHP? JSP? what about web.xml 
descriptors? are they any better?

Second point: how in hell is "structure of the URL space" different from 
"the URL space of the application"?

Third point: this is *flow* is should *NOT* be part of a sitemap anyway.

> The basic view of a URL in the sitemap is that it is any string. Even if 
> there are constructions like mount, the URL space is not considered as 
> hierarchial. That means that the URLs can be presented as patterns in 
> any order in the sitemap and you have to read through all of it to see 
> if there is a rule for a certain URL.

As I mentioned already, this is a design decision based on the fact that 
it is *arbitrary* to consider the / as a hierachical separator.

Also, matchers are *NOT* URL-specific and it's a very useful concept. 
Forcing matching to be:

  1) URL-based


  2) intrinsically hierarchical

is IMO a *severe* step backward in terms of architectural design.

> A real map for the site should be tree structured like the linkmap in 
> forrest. Take a look at the example in [1], (I don't suggest using the 
> "free form" XML, something stricter is required). Such a tree model will 
> also help in planning the URI space as it gives a good overview of it.

Forrest and cocoon serve different purposes.

While I totally welcome the fact that Forrest has such "linkmaps", I 
don't think they are general-enough concepts to drive the entire 
framework. They are fine as specific cases, especially appealing for a 
website generation facility like forrest, but as a general concept is 
too weak.

> The Forrest linkmap have no notion of wildcards, which is a must in 
> Cocoon. We continue discussing that.

All right.

> Choosing Production Pipeline
> ----------------------------
> With the sitemap it is very easy to choose the pipeline used for 
> producing the response based on a URL pattern "*.x.y.z". That more or 
> less forces the user to code implementation details i.e. what pipeline 
> to use into the URL. This is only a problem for wildcard patterns 
> otherwise we just associate the pipeline to the concrete "cool URL".

At this point I seriously wonder: are you aware that matchers are pluggable?

> Before I suggested that aspects like: type, format, status, access 
> rights etc shouldn't be part of the URL as those aspects might change 
> for the resource. OTH these aspects certainly are necessary for choosing 
> rendering pipeline, what should we do?

URL-parameter matching.

  <match type="wildcard" pattern="/news/*">
    <match type="param" pattern="edit">
    <match type="param" pattern="delete">

or, if you have HTTP action control (as in form actions), you can do

  <match type="wildcard" pattern="/news/*">
    <match type="action" pattern="get">
    <match type="action" pattern="post">

and, most of all, you do *NOT* include access control information in the 
URL! nor type! nor status!

> The requested resource will often be based on some content or 
> combination of content that we can access from Cocoon. The content can 
> be a file, data in a db, result from a business object etc. Let us 
> assume that it resides in some kind of content repository. Now if we 
> think about it, isn't it more natural to ask the content, that we are 
> going to use, about its propertiies like type, format, status, access 
> rights, etc, than to encode it in the URL? This properties can be 
> encoded in the file name, in metadata in some property file, within the 
> file, in a DB etc. 

Ok, now that the nonsense venting is over, we seem to be getting at your RT.

> Now instead of having the rule:
> *.x.y.z ==> XYZPipeline
> we have
> * where repository:{1} have properites {x, y, z} ==> XYZPipeline
> or
> * where repository:{1}.x.y.z exists ==> XYZPipeline

Oh, a rule system for sitemap!

hmmmm, interesting... know what? the above smells a *lot* like you are 
querying RDF. hmmmm...

> We get the right pipeline by querying the repository instead of encoding 
> it in the URL. A further advantage is that the rule becomes "listable" 
> as the "where" clause in the match expresses what are the allowed values 
> for the wildcard.
> Separating the Concerns
> -----------------------
> The sitemap handles two concerns: it maps an URL to a pipeline that 
> produces a repsonse and it describes how to put together this pipeline 
> from sitemap components.


> The first concern is related to site design and 
> the second is more a form of programming. Puting them together makes it 
> hard to see the URL structure and also makes it tempting to group URLs 
> based on common pipeline implementation instead of on site structure.

Fair enough.

> Virtual Pipeline Components (VPCs) give us a way out from this. Large 
> parts of our sites might be buildable with pipelines allready 
> constructed in some standard blocks.


> I would propose to go even further, in the "real" site map it should 
> only be allowed to call VPC pipelines, no pipeline construction is 
> allowed, that should be done in the component area.
> In the "real" site map the current context is set and the the arguments 
> to the called VPC is given.

Hmmm, rather drastic, but let's stick to it for your proposal.

> Search Order
> ------------
>>  The problem for us, is as you allude to at the start of this
>> thread: Cocoon takes the first match, where what you really want is a
>> more XSLT "best match" type of handling; sometimes *.a, *.b, *.c works
>> and other times it's m.*, n.*, o.*...
>> In the past that has lead me to suggest a sort of XSLT flow, but
>> thinking about it in this light I wonder if what I really want is just
>> XSLT sitemap matching (same thing in the end)...
> I also believe that a "best match" type of handling is preferable, it 
> increases IMO usabillity and it also makes it possible to use tree based 
> maching algoritms that are far more efficient than the current linear 
> search based.

This is a valid point.

> The new sitemap
> ===========
> To sum up the proposal:
> Pipelines:
> * Pipeline construction is only done as VPCs in component areas (often 
> in blocks).
> Sitemap:
> * The sitemap is folow the tree structure of the URL space (like the 
> Forrest linkmap).
> * Its responsibillity is to map URLs to VPCs
> * It can set the current context for each level in the tree (for 
> derefering relative paths used in the VPC)
> * Wildcards can have restrictions based on properties in the content 
> repository
> * Its best match based rather than rule order based
> * Of course we have an include construct so that we can reuse sub sites
> It might look like:
> <sitemap>
>  <path match="person" context="adm/persons" 
> pipeline="block:skin:default(search.xml)">
>    <path match="*:patient" test="mydb:/patients/{patient} exists" 
> context="adm/patients" pipeline="journal-summary({patient})">
>      <path match="edit" pipeline="edit({patient})"/>
>      <path match="list" pipeline="list({patient})"/>
>      <!-- and so on -->
>    </path>
>  </path>
> </sitemap>
> Don't care about the syntactical details in the example it needs much 
> more thought, I just wanted to make it a little bit more concrete. The 
> path separator "/" is implicily assumed between the levels. "*:patient", 
> means that the content of "*" can be refered to as "patient".
> Much of what I propose can be achieved with VPCs and a new "property 
> aware" matcher. But IMO the stricter SoC above, the ability to "query" 
> the sitemap, the possible advantages of the "best match" search, are 
> reasons enough to go further.

First thing that comes to mind is that the implicit assumption of '/' is 
just bad. I would be against the proposal just for that.

Second, you lose the ability to do non-URL matching, which is, again 
another reason to vote against this.

Third, conditional matching is just nonsense, it's mixing flow concerns 
with matching.

Forth, I don't find the above any more readable than a sitemap that uses 

I'll think about the rule-based pipeline resolution (which is an 
interesting concept on itself) but the rest, I'm sorry, it really does 
not resonate with me at all.


View raw message