Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (asf.osuosl.org: domain of lists@holsman.net designates
 218.214.209.234 as permitted sender)
Message-ID: <42D59A72.2060805@holsman.net>
Date: Thu, 14 Jul 2005 08:49:22 +1000
From: Ian Holsman <lists@holsman.net>
User-Agent: Mozilla Thunderbird 1.0+ (Macintosh/20050529)
MIME-Version: 1.0
To: dev@httpd.apache.org
Subject: Re: mod_mbox: js speed and mbox size
References: <20050713195102.GA9780@bulix.org> <42D57762.10804@wstoddard.com>
In-Reply-To: <42D57762.10804@wstoddard.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Bill Stoddard wrote:
> Maxime Petazzoni wrote:
> 
>> Hi,
>>
>> As I already said, the AJAX browser in mod_mbox relies on a hidden
>> XmlHttpRequest and on Javascript processing of the replied XML
>> document to generate the message list.
>>
>> After setting up a browser stub, I'm not going to concentrate on
>> compatibility and speed problems before continuing. In fact, there are
>> two main problems implied by the current mechanism :
>>
>> - On large mailboxes, the /ajax/msglist query reply can get *very*
>>   large. As an example, this mailing list's 2001-08 archives contains
>>   1708 messages, and the corresponding XML is 452492 bytes big. This
>>   is too much. As a comparison, the old mod_mbox resulting page for
>>   the same month is "only" 311 KB big.

mod_deflate should be able to help with that. XML (or any text) 
compresses real well.

>>
>>   As a solution to this, Ian advised the use of LiveGrid
>>   (http://openrico.org/livegrid.page). It fetches only the piece of
>>   data that need to be displayed (ten, twenty, ... rows instead of the
>>   whole 1700+). I don't really like this solution because the user
>>   must scroll the table, and then wait for the data to appear,
>>   etc. 
> 
> Isn't this where XmlHttpRequest comes in handy?  Asynchronously fetch X 
> (a hundred?) rows forward from where the user is currently looking in 
> anticipation that that user will scroll forward. So you don't slurp down 
> the whole mailbox at once, but you try to stay ahead of what the user 
> might be expected to do and always have some data ready to display.
> 

That is what openrico do I'm guessing. they just provide a set of higher 
level API's to manage that.

>> And that we don't get the whole list (no in-page search for
>>   example).
>>
>>   Another solution would be to use a page system as in the Debian
>>   mailing list archives, displaying 100 or 200 messages per page
>>   only. This would reduce bandwidth usage, but it will also make things
>>   harder when updating the context box. For the moment, the message
>>   list is fetched once, and all operations are made on this stored
>>   list. When updating the context box, we just take -3 to +3 messages
>>   from the list. With the page system, we'll experience problems while
>>   approaching the end of a page ...
>>
>> - The second problem is brought by the first one. On large mailboxes,
>>   the Javascript processing takes too much time. Building all elements
>>   each time the message list is displayed (at loading time or when you
>>   close a message box) does not make the application usable.
>>

would preprocessing (and caching) some of this stuff on the server help?
it might make your 450k XML file larger, but you could provide some of 
the thread info in the XML file itself.

>>   Of course, the paging system would solve this problem, too. In
>>   addition to this, my Javascript code may not be the most optimized
>>   we could find.
>>
>> - Concerning compatibility problems, the whole thing does not work
>>   nor with IE, neither with Opera, although both of these browsers
>>   does not comply against any javascript error.
>>
>>   Any JS compatibility expert in the house ?

yes. I've done a bit, and it is yuk

this is why I suggested openrico .. they have done all of this for you, 
and have their code used by a wider range of people. the code has a ASL2 
licence.. it is perfect for httpd.

> 
> Heh, you wish :0
> 
> This is a cool project, glad it's on the httpd dev list.
> 
> Bill
>