camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <>
Subject Re: [HEADS UP] Camel Web Site changes....
Date Fri, 09 Mar 2012 11:27:05 GMT

Nice that you picked up this glove, and got started early in the year,
so we wont have a last-minute issue when the year is closing.

On Wed, Mar 7, 2012 at 6:54 PM, Daniel Kulp <> wrote:
> As some of you are aware, the Apache Infrastructure team has mandated that
> all projects move to the new svnpubsub process for publishing their websites
> by the end of the year.   Camel (as are all Confluence based sites) is
> affected by this mandate.   We currently use a multi-step rsync process
> where Confluence exports the space to HTML, that gets rsynced to an area on
> once an hour.   A cron process in someone's crontab on
> then runs to rsync it to the appropriate place
> (/www/ once an hour or so as well.   Then, another rsync
> will sync from there to the live sites.    This causes a lot of delays in
> publishing (can be a few hours between change and live), but also involves a
> LOT of disk IO to sync things all over the place.   The svnpubsub process is
> a lot faster as the site changes are committed to svn and anything that
> needs it (the live site) can listen for the changes and update immediately.
> Anyway, over the last couple of months, a bit of work has been done with
> various projects to start helping projects transition to svnpubsub.  Joe
> Schaefer has been working with the Maven folks so the "mvn site:deploy"
> stuff can deploy via svnpubsub.   Obviously the CMS uses it heavily.
> We also now have a "solution" for Confluence based sites based on work I've
> done for CXF's site.    Confluence has a SOAP interface for retrieving
> information and rendering pages.    Well, if I see a SOAP interface (even a
> crappy one like Confluence's)......  ;-)
> Seriously, I have a program now that can render an entire confluence space
> by using the SOAP API's and Velocity (which is what the current AutoExport
> stuff uses, so migration is easy).   However, it does more than that by also
> recording the modified times, checking the RSS feed for changes first,
> tracking {children} and {include} tags, etc...   Thus, if you change a page
> that is "included" in another (think about the "Book in One Page" page),
> those pages will also get re-rendered.    If you add/delete a page, any page
> that uses the {children} tag to generate a tableof contents will
> automatically re-render.
> It ALSO cleans up the resulting HTML via tagsoup and some custom cleanup
> code.  The Confluence generated HTML is aweful with invalid attributes, bad
> links, etc...   They are now "mostly" cleaned up.
> I've uploaded a "build" of the site to:
> so you can see that the result is pretty much identical to the live site.
> A couple of things are actually better such as the image links for the blog
> entries.   Also, the new page actually validates with the w3c validator:
> To run this, a buildbot build will be setup to run the process once an hour
> to generate new html if it detects a change (rss feed).  Once run, you get a
> commit message to the commits list and the changes are live immedately.
> Thus, changes are now "at most" one hour till they are on the live site.
> However, any commiter can checkout the stuff and run it manually if they
> need/want things live immiately.
> For CXF, this new process is now "live" (since Monday).   I've filed a
> ticket with INFRA to start the process for Camel. (requires a content area
> in the web svn repo for the live content, then a buildbot build, then some
> configs to make it all live)    It's definitely still a "work in progress",
> but it's a good start.   For example, it doesn't track the blog/news entries
> so currently if you add a blog (for a release), you would need to manually
> trigger an Index page update.   However, the code is there so it's something
> we can add/enhance.  I also want to update it to render pages in parallel if
> possible to make it a bit quicker.
> For camel, the main "pom" and scripts are in:
> and there is a README there.   The code for the stuff is grabbed via an
> svn:externals to the area in CXF so I just need to update the code in one
> place:
> I want to avoid "forking" the code as I *DO* know that if/when they move to
> Confluence 4.x (currently on 3.4.x), it will need some updating as the SOAP
> API's changes a bit.
> Anyway, I'm hoping to have Camel flipped over by the end of the week or
> early next week.
> BTW:  if you are interested, it has to render 644 pages for the full Camel
> website.   Takes about 15 minutes to do right now, but like I said, once
> it's all setup, it can do incremental updates which is MUCH quicker.  644
> pages is quite a bit.
> --
> Daniel Kulp
> -
> Talend Community Coder -

Claus Ibsen
Twitter: davsclaus, fusenews
Author of Camel in Action:

View raw message