forrest-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicola Ken Barozzi <>
Subject New CLI and Speed (was Re: residuals of MIME type bug ?)
Date Thu, 26 Jun 2003 17:34:05 GMT

Nicola Ken Barozzi wrote, On 26/06/2003 10.20:
> Upayavira wrote, On 26/06/2003 9.43:
>> There are quite a lot of new features in the Cocoon CLI that Forrest 
>> isn't using, for example the option to switch off mime-type checking, 
>> and to only scan pages once (i.e. not using the link-view) to follow 
>> links.
> We have to use them, I agree. In fact Forrest was the primary use case 
> of having more speed with the new CLI and for not using the mimetype thing.

Oh, and also not having error pages generated where there are errors, so 
that a link checker run on live sites can see the real broken links.

Speed comparison with the Krysalis Version project site:

   - standard Forrest:   57 seconds
   - with -etrue option: 27 seconds

That's double speed. And memory usage is lower too.
It also *feels* much faster, which is not to forget too.

>> I believe there are still some problems with these new features in the 
>> CLI, but it should be possible to fix these. [For example, links being 
>> gathered on pipelines referenced via cocoon: protocol - I've found 
>> why, but not yet fixed it].

In fact the Forrest site cannot be generated with this CLI method, as it
does not play well with link rewriting because of the above bug. I guess
it's because you insert the gatherer *before* the rewriting, because of
the cocoon: protocol usage IIUC as you say. The wierd thing is that the 
page gets rendered right, I guess it's just the cocoon:-called pipeline 
that complains.

Then there is again the recursion bug, that makes links get gathered in 
a recursive manner, making them longer and longer and longer...

IIRC we had this before, wasn't it already fixed?

>> Is anyone interested in looking into how to upgrade Forrest to use 
>> these new features?
>> I think that doing this would stand a chance of resolving all of Luc's 
>> problems, and give me some people to do some solid debugging of the CLI.

To enable this method, users just need to do this:

  - go in the dist/shbat dir
  - edit
  - insert the following line in the Cocoon args:
     <arg value="-efalse"/>

Nicola Ken Barozzi         
             - verba volant, scripta manent -
    (discussions get forgotten, just code remains)

View raw message