incubator-odf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Tika is waiting for ODFToolkit to improve ODF file format processing
Date Tue, 25 Oct 2011 17:03:40 GMT
On Mon, Oct 24, 2011 at 9:17 AM, Rob Weir <robweir@apache.org> wrote:
> On Mon, Oct 24, 2011 at 4:54 AM, Devin Han <devinhan@apache.org> wrote:
>> I saw this issue in Tika: OpenOffice parser: master footer text isn't
>> extracted https://issues.apache.org/jira/browse/TIKA-736
>>
>> The current ODF parser of Tika doesn't touch the styles part and the embeded
>> document, only meta and content. They are waiting for the first ODF Toolkit
>> incubating release, then switch to a full featured parser much as they have
>> for the POI powered ones.
>>
>> The first release is coming and we will have no code update before it. So, I
>> suggest start the discussion that how to use ODF Toolkit to realize it based
>> on the snapshot.
>>
>
> In that JIRA thread Uwe talks about the desire for a
> streaming/SAX-like API for scanning the ODF documents.  I agree.  The
> DOM approach we use with ODF Toolkit is necessary for when you need
> random, read/write access to a document.  But you pay a performance
> (mainly heap memory) penalty for that flexibility.  But if you can
> organize your program logic into a single-pass read-only approach,
> then a streaming approach can -- in theory -- perform much better for
> that restricted use case.  But I still wonder how much the underlying
> ZipInputStream implementation actually manages to stream the deflate
> algorithm when it unzips ODF's ZIP package....
>
> In any case, this is something I'd be interested in working on after
> we get our initial ODF Toolkit release out.  A memory optimized
> streaming API for read-only, single pass uses.

I agree a more SAX-like (single pass, don't hold stuff in RAM)
approach would mostly fit Tika's needs well.

Note that the DOM approach is also used by other parsers Tika wraps
(eg PDFBox, POI I think), so this is not a "unique" challenge for
ODF.

Tika's needs are actually quite simple compared to what ODFToolkit can
do.

Ie, really we just need read-only single pass (document -> text), with
some amount of document structure retained (so we know where to put
<p>, <div>, <b>, etc., tags).

For TIKA-736 in particular, it'd be nice to "reconstruct" each slide
so that any text from the master slide/layout is inlined into each
slide that uses it, so that the resulting text looks the way it looks
when you view the document in OpenOffice.  This is the approach we're
working towards in TIKA-712 for PPT/X files.

I imagine to do this you'd need DOM-like access to the master slide /
layout / style, and could then us SAX-like single pass for the
"normal" slides.

TIKA-735 is another issue with the the current ODF parser, whereby the text
from embedded documents is always placed at the end of the text from
the original document, rather than being inlined at the point where
the embedding occurred.  Seems like a SAX like API would work fine
here, ie, we should simply recurse into the embedded doc when we
encounter it.

Mike McCandless

http://blog.mikemccandless.com

Mime
View raw message