cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Quinn <>
Subject Re: request encoding conundrum
Date Tue, 29 Jul 2008 11:29:39 GMT
Hi Grzegorz

Gosh! A lot to respond to :)

On 27 Jul 2008, at 18:51, Grzegorz Kossakowski wrote:

> Jeremy Quinn pisze:
>> I still have all of the notes and the builds we did (thanks!).
>> But I am still doing the work in 2.1, as (if I remember properly)  
>> we did not manage to make a build that would edit live at the level  
>> of the cforms block itself.
>> Correct me if I am wrong, but it seems easier to setup 2.1 so that  
>> edits made to the built-in resources of the block are immediately  
>> live without re-building.
> Jeremy, yep, you are wrong here. ;-)

Glad to hear it :)


> I hope that this is impressive enough way of convincing you that you  
> are wrong. ;-)

Thanks for your effort !

I'll work through that lot to see if I can adapt it to the scheme you  
helped me with to work without Eclipse ..... which as you know I only  
use if absolutely necessary ....

>> ... but I am still bogged down with subtle differences in format  
>> interpretation between Java and Dojo, with validating number  
>> fields, it's a minefield ...... blog entry half written ;)
> Eager to read.

Still working out the details of what goes wrong ....

>>> I see only two small obstacles:
>>> 1. As I have already seen it at ApacheCon you have some nice work  
>>> in your computer. The problem is that if you keep it on your  
>>> computer then nobody can test it and eventually help you with this  
>>> stuff. Any reason to not commit your work that you already have to  
>>> some public place?
>> There are a few problems that have stopped me doing this so far :
>> 1) too lazy (so far) to set up and maintain some kind of branch/ 
>> sandbox ;)
> Weak excuse I must say, branching blocks (at least in 2.2) is a very  
> easy task. Actually, we have maintenance branches for Forms and  
> Template blocks here:
> I could setup branches for Dojo playings there if you wish.

For sure

>> 2) I cannot commit anything to head yet, because lots of stuff is  
>> still completely broken and/or still has to be re-written to the  
>> new APIs. The work has already taken me several months, and there  
>> are several more to go ..... it is unpredictable how much longer  
>> this will take, I'd mess up Cocoon's release cycles .....
> Yes, that's why branching is the only option.


>>> Otherwise any collaboration is rather difficult.
>> Agreed.
>> What would you propose?
>> The work involves having two or three custom blocks, forms and ajax  
>> (atm, I have dojotoolkit as a block).
>> If you are serious about getting involved, I'd be prepared to make  
>> the extra effort to collaborate.
> It's difficult to say if I'm serious because my plans are little bit  
> changeable at the moment. The situation looks like this: The company  
> I'm working at is seriously interested in migrating Forms to Dojo  
> 1.x but now we are busy with migration to 2.2 that nobody knows how  
> long will take. Actually, we have some estimations and provided  
> everything goes well I'll be working on this migration *very* soon.


> On the other hand, we are making an open source here, right? I would  
> much more prefer you commiting small changes that others can review  
> than coming with big contribution that nobody can check or follow.

It's catch 22
As soon as I replace dojo 0.4 with 1.1 EVERYTHING broke, so the  
changeset is huge. There is no way around this .... OK, so I am  
compounding that by adding new features as I go, but basically yes,  
there will be some massive commits ....

> Moreover, I've seen some users playing with Dojo 1.x and Cocoon 2.2  
> on our users mailing list so there are chances that you will get  
> some supporters even if my plans change for some reason.
> Therefore, I would suggest to publish your work regardless of my own  
> plans.

For sure

>>> 2. I prefer to work with C2.2 (trunk) because it's simpler than  
>>> 2.1 and it's much easier to develop/test anything here. Any  
>>> chances that you will switch with your work to trunk?
>> You find 2.2 simpler, I find 2.1 simpler :)
>> If we could find the right way to collaborate, you can work on 2.2- 
>> specific issues, and I can work on 2.1.
> Jeremy, I hope that my video will convince you that things with 2.2  
> are simpler. Another fact is that more and more people interested in  
> Cocoon switch to 2.2 so your work on it will get more wider audience.
> Also, take into the account that in 2.2 we can release blocks  
> independently of the rest so it's much easier release cycles with  
> real work that gets done.
>> One of the major problems with 2.2 is the loss of the 'system  
>> pipelines' that in 2.1 provide a set of static URIs for loading  
>> cforms and dojo resources; coupled to the fact that /someone/  
>> misunderstanding dojo APIs thought it necessary to introduce a  
>> resource-path for use by cforms widgets client-side.
> It's been me who removed 'system pipelines' in trunk as we have a  
> superior mechanism for serving block resources: it's servlet:  
> protocol and Servlet Service Framework in general.

Yes, and it's OK, I understand why ...

> When it comes to the resource-path it wasn't me who introduced this  
> variable.

Don't worry, I know that :)

>> I can hopefully help you over-come these problems.
>> This is the current JS Loader for 2.1.12-dev :
>> <script src="/_cocoon/resources/dojotoolkit/dojo/dojo.js"  
>> type="text/javascript" djConfig="isDebug: true, locale: 'en_GB',  
>> parseOnLoad: true"></script>
>> <script type="text/javascript">
>> dojo.require("dojo.parser");
>> dojo.registerModulePath("cocoon.forms", "../../forms/js");  
>> dojo.registerModulePath("cocoon.ajax", "../../ajax/js");  
>> dojo.require("cocoon.forms.common");  
>> dojo.addOnLoad(cocoon.forms.callOnLoadHandlers);
>> </script>
>> (ignoring paths to css for now ....)
>> We have a system pipeline "/_cocoon/resources/ .... " which is used  
>> as a prefix to load dojo from the dojotoolkit block.
> Yep, I remember how it was done in C2.1. Now let's have a look at  
> the same fragment in 2.2:
> <script src="{$dojo-resources}/dojo.js" type="text/javascript"/ 
> >           <!-- load dojo -->
> <script type="text/ 
> javascript">                                            <!-- load  
> forms library -->
>  <xsl:value-of select="concat('dojo.registerModulePath(',  
> $doubleQuote, 'cocoon.forms', $doubleQuote, ', ', $doubleQuote,  
> $forms-resources, '/js', $doubleQuote, ');')"/>
>   <!-- tell dojo how to find our forms module. NB: (since 2.1.11,  
> replaces cocoon.js) -->
>    dojo.require("cocoon.forms.common");  <!-- tell dojo we require  
> the commons library -->
> Apart from the fact that there is an ugly concation these snippets  
> look similar.


>> Then we register two modules, forms and ajax, using a path that is  
>> relative to where dojo was loaded from.
> The problem is that in 2.2 every block can have an arbitrary  
> location where it's mounted so relative paths across different  
> blocks won't work. This is a consequence of the fact that block can  
> be *extended* and extending block must have different mount path.
> Cocoon 2.2 is less monolith so you can make less number of  
> assumptions but fortunely enough there is one thing coming to yours  
> rescue: it's servlet protocol.
> You can just write:
> <script src="servlet:dojo-block:/path/relative/to/dojo.js"  
> type="text/javascript"/>
> Where this path is relative to block's root directory and rest will  
> be handled by Cocoon. The nice thing about using servlet: protocol  
> is that it's aware of inheritance. So for example, you can create  
> your-fany-dojo-block and declare dojo-block as a super block for it.  
> Then you can override just a single resource (like CSS) and servlet:  
> protocol will handle it.
> Actually, you don't need to use servlet: protocol everywere. It's  
> enough if you use it to obtain mount path for the block. So there is  
> no trouble with calculated paths in Javascript, etc.


So how/when/where does 'servlet:forms-block:', 'servlet:ajax-block:',  
'servlet:dojo-block:' etc. get resolved to a URI for the browser?

> All these things influence how blocks in 2.2 look like but Servlet  
> Service Framework is such a killer feature that it's simply worth  
> small inconveniences.

It would be great to 'hide' the complexity of the current forms- 
processing pipeline. It is long and complex and does not need to be  
touched by most users.

Would 2.2's much vaunted 'virtual pipelines' help here?

This is the current JX Forms rending pipeline (2.1.12-dev,  
uncommitted) with notes (##) to describe where custom parameters need  

<map:match pattern="*-display-pipeline.jx">
     <map:generate type="jx" src="forms/{1}_template.xml"  
         <map:parameter name="locale" value="{flow-attribute:locale}"/>
     <map:transform type="browser-update"/><!-- AJAX-style form update  
     <map:transform type="i18n">
         <map:parameter name="locale" value="{flow-attribute:locale}"/>
<!-- ## call the User's XML-->HTML transformation resource here  
(instead of simple-page2html) -->
     <map:call resource="simple-page2html">
         <map:parameter name="file" value="forms/{1}_template.xml"/>
     <map:transform src="resources/forms-samples-styling.xsl">
<!-- ## inject references to block locations? -->
         <map:parameter name="resources-uri"  
<!-- ## allow User to turn debugging on and off -->
         <map:parameter name="dojo-debug" value="true"/>
         <map:parameter name="dojo-locale" value="{flow-attr:locale}"/>
<!-- ## probably only wanted in samples -->
         <map:parameter name="dojo-theme-param" value="{request- 
<!-- ## allow User to choose default theme -->
         <map:parameter name="dojo-theme-default" value="tundra"/>
     <map:transform type="i18n">
         <map:parameter name="locale" value="{flow-attribute:locale}"/>
     <map:select type="ajax-request">
         <map:when test="true"><!-- sending a partial Browser Update -->
             <map:select type="request-parameter">
                 <map:parameter name="parameter-name"  
                 <map:when test="iframe"><!-- sending BU via iFrameIO  
                     <map:transform src="resource://org/apache/cocoon/ 
                     <map:serialize type="xhtml"/><!-- this is REALLY  
important -->
                 <map:otherwise><!-- sending BU via XHR -->
                     <map:serialize type="xml"/>
         <map:otherwise><!-- sending a complete page -->
             <map:serialize type="html"/>

BTW. Do virtual pipelines work without the intermediate serialize/de  
serialize step yet?

>> One point that was missed by the /someone/ above, was that once a  
>> module is registered, you can get a url to it like this :
>> var imgSrc = dojo.moduleUrl("cocoon.forms","images/blah.png");
>> i.e. it is not necessary to provide it specifically to the client  
>> as it is currently done : cocoon.resourcesUri = "<xsl:value-of  
>> select="$resources-uri"/>"
> As I said earlier, I can agree with this only if you access  
> resources from one block. Anyway, usually that's the case.
>> But TBH, except for a few exceptions like custom data-source urls  
>> (dynamic selectionlists etc.) there should be no need to reference  
>> anything like this ..... templates should be embedded in widgets,  
>> images used in widgets should be loaded via css (where relative  
>> references work internally) etc. etc.
> Agreed.
>> So, the system path is not available in 2.2. The dojotolkit, forms  
>> and ajax blocks could have any URI. So we need a standard way for  
>> an application block to tell it's form-rendering pipeline the paths  
>> to these blocks. Presumably this should be the responsibility of  
>> the application's sitemap.
>> It should not be necessary to re-write any URIs (!!).
> Unfortunately, it's necessary due to reasons I outlined above. But  
> if these things are handled in 100% transparent way then is there  
> any problem?

When you talked about re-writing URIs, I thought you meant post- 
processing urls in xml or html, or grepping javascript files (!).

Now I am not sure that was what you meant ..... I think actually you  
are just assembling URIs, once, from supplied parameters, this is not  
a problem IMHO.

>> Furthermore, this provision of paths to blocks, needs to take into  
>> account the fact that in production people will most likely want to  
>> do stuff like :
>> 1) acquire dojo from CDNs like AOL, Google etc.
> This is more interesting. To be honest, I have no idea how this  
> could be *easily* handled by 2.2.

I don't even know if this /works/ yet :)

>> 2) build custom minimised JS libs to support their apps
>> 3) load their own custom modules, override css etc.
> This is a damn easy thing in 2.2. That's why I see development on  
> top of 2.2 /easier/ compared to 2.1.


>> 4) lots of stuff we have not thought of yet ;)
>> ATM, while I am developing cforms, my dojotoolkit block is a  
>> special build, everything uncompressed, unpackaged, etc. with like  
>> 180 sets of locales etc. etc. Some complex forms are loading over  
>> 100 separate assets.
>> The modularity of dojo (and by using dojo.require) means that only  
>> what is needed by a page is loaded, which is great. But in  
>> production, you will want to heavily reduce the number of  
>> files ..... specially the 404s you get 'hunting' the locale tree.  
>> It is a bit of a contradiction .....
> I don't really follow you here. Could you explain the fragment about  
> 404s and 'hunting' the locale tree?

Very similar to the way the i18nTransformer works.
are optional).
Message files are (sparsely populated) in nested folders :


When i18n is being resolved, it hunts through this hierarchy looking  
for the most specific match.
The i18nTransformer does this on the server.
Dojo also works the same way, but has to make HTTP Requests to look  
for stuff.
As the hierarchy is sparse, it may legitimately trigger 404's as it  
looks for the best match.

>> I have not really begun to think seriously about how this should be  
>> done yet.
> In 2.2 I can see that all these things are done following way:
> We ship reasonably standard dojo/ajax block on which Forms block  
> depend on. When someone wants to customize it by overriding  
> resources or by reducing number of files he creates his custom dojo  
> block that *extends* our standard dojo block. Then one can override  
> any pipeline from standard dojo block serving resources by it's own  
> implementation. One can override pipeline's behavior only for some  
> resources and fall-back to standard block for the rest.
> The last step is to tell Forms to use customized dojo block which is  
> easy and involves setting one property so original Forms block don't  
> have to be touched at all.

Sounds great!

> All these magic is already implemented and works well in 2.2 but  
> sadly enough most of people seem to be not aware of it. The reason  
> is rather simple: we don't provide any good sample showing how this  
> works. I guess this is something that should be fixed quickly.

This should definitely happen IMHO.
Anyone new to Cocoon who looked at the current site, would probably be  
left with the impressions that it was designed primarily as a  
container for Spring Beans :) :) :)

>> If we could collaborate on a way to cleanly solve this, so that  
>> ideally the basic technique is the same for both 2.1 and 2.2, that  
>> would be really useful for me :)
> You probably already got the impression that I try to convince you  
> to forget about 2.1 and switch to 2.2. :)
> I wholeheartedly believe this is a right thing to do but I'm not  
> sure about your feelings.

I think we have discussed this already :)
I fully intend to provide these changes to 2.1. I still know many  
projects using 2.1 and see no reason why it should be orphaned. IMHO  
this perceived conflict between 2.1 and 2.2 has already done the whole  
project a lot of damage, I do not wish to contribute to that.

When my work is completed for 2.1, I intend to apply the same to 2.2  
(unless someone else kindly does it for me :)

> Unfortunately, Forms block in 2.1 and Forms block in 2.2 are rather  
> far away from themselves and this distance is going to be bigger but  
> not smaller in the future.

The differences seem to be primarily with the sitemap.
The Java Classes, JavaScript and XSLT, should be identical, no?

>>> I've tested it (combined with fix from COCOON-1917) and on the  
>>> server side everything looks correct now.
>> Great !!!
>>> The only problem is that browser sometimes does not behave  
>>> correctly.
>>> I noticed that sometimes when I enter non-latin characters to the  
>>> text field they get escaped by a browser.
>>> So when I enter something like:
>>> światło
>>> the browser posts to the server such value:
>>> &#347;wiat&#322;o
>> Yes, I see this a lot.
>> I also see UTF-8 encoding like this : %E2%82%AC (which is the 3  
>> byte encoding for the Euro symbol).
>> I have not found this encoding to be a problem.
> I thought that browser should never escape characters if it's not  
> absolutely necessary. If browser was submitting the data using UTF-8  
> then that wouldn't be necessary right?

Browser are free to do what they like .....
They may or may not choose to encode .... what is important is that  
encoding is (a) correct, (b) properly interpreted where needed (this  
is what my patch was to address).

>> What problem does this cause you?
> Our samples are simply broken that's the problem :-)

That statement contained zero information :)

> If you try to use this upload sample I've already pointed you to  
> then you will see that result page produced after forms is submitted  
> contains escaped characters.

I will have a look at that sample ASAP.
But first I have to re-write the upload widget to work with dojo  
1.1.1 :)

>>> (additionally there is parameter: dojo.transport=xmlhttp)
>> This is one of the standard parameters that CForms has to add to  
>> form submits.
>> CForms uses 3 different transports, depending on context:
>> 1) ajax-off : normal whole page submit
>> 2) ajax-on  : xmlhttp
>> 3) ajax-on + form contains a 'file' field : iframe-transport
>> Unfortunately, the response to each of these needs to be serialized  
>> differently, hence the need to a very complicated sitemap for  
>> cforms and this special parameter.
> I see. Still I don't understand why our samples are broken.

My understanding is that it is due to browsers beginning to do the  
right thing, i.e. inform the server what encoding is being used. The  
fault lay in Cocoon not realising it had been told, and therefore  
making wrong assumptions.

>>> Since I don't know how these things are handled on the client side  
>>> I'm not sure how to fix it.
>>> Any ideas?
>> I need more details of what problem it causes ....
> I hope you can reproduce it with upload sample we have in Cocoon.

I'll let you know ....

Many thanks for your feedback etc. Grzegorz.
Please face up to the fact that I want this to work with 2.1 as well  
as 2.2 :)
It may be more adaptable in 2.2, but that does not mean I want to  
leave 2.1 behind.

best regards


View raw message