incubator-odf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Weir <robw...@apache.org>
Subject Re: Tika is waiting for ODFToolkit to improve ODF file format processing
Date Tue, 25 Oct 2011 21:40:56 GMT
On Tue, Oct 25, 2011 at 1:03 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> On Mon, Oct 24, 2011 at 9:17 AM, Rob Weir <robweir@apache.org> wrote:
>> On Mon, Oct 24, 2011 at 4:54 AM, Devin Han <devinhan@apache.org> wrote:
>>> I saw this issue in Tika: OpenOffice parser: master footer text isn't
>>> extracted https://issues.apache.org/jira/browse/TIKA-736
>>>
>>> The current ODF parser of Tika doesn't touch the styles part and the embeded
>>> document, only meta and content. They are waiting for the first ODF Toolkit
>>> incubating release, then switch to a full featured parser much as they have
>>> for the POI powered ones.
>>>
>>> The first release is coming and we will have no code update before it. So, I
>>> suggest start the discussion that how to use ODF Toolkit to realize it based
>>> on the snapshot.
>>>
>>
>> In that JIRA thread Uwe talks about the desire for a
>> streaming/SAX-like API for scanning the ODF documents.  I agree.  The
>> DOM approach we use with ODF Toolkit is necessary for when you need
>> random, read/write access to a document.  But you pay a performance
>> (mainly heap memory) penalty for that flexibility.  But if you can
>> organize your program logic into a single-pass read-only approach,
>> then a streaming approach can -- in theory -- perform much better for
>> that restricted use case.  But I still wonder how much the underlying
>> ZipInputStream implementation actually manages to stream the deflate
>> algorithm when it unzips ODF's ZIP package....
>>
>> In any case, this is something I'd be interested in working on after
>> we get our initial ODF Toolkit release out.  A memory optimized
>> streaming API for read-only, single pass uses.
>
> I agree a more SAX-like (single pass, don't hold stuff in RAM)
> approach would mostly fit Tika's needs well.
>
> Note that the DOM approach is also used by other parsers Tika wraps
> (eg PDFBox, POI I think), so this is not a "unique" challenge for
> ODF.
>
> Tika's needs are actually quite simple compared to what ODFToolkit can
> do.
>
> Ie, really we just need read-only single pass (document -> text), with
> some amount of document structure retained (so we know where to put
> <p>, <div>, <b>, etc., tags).
>

Is there a list of the complete set of tags you use, or a schema or something?

> For TIKA-736 in particular, it'd be nice to "reconstruct" each slide
> so that any text from the master slide/layout is inlined into each
> slide that uses it, so that the resulting text looks the way it looks
> when you view the document in OpenOffice.  This is the approach we're
> working towards in TIKA-712 for PPT/X files.
>

Text box position is ultimately encoded as x,y coordinates on the
slide.  So the visual appearance on the slide and the order of the
text boxes in the document's XML are generally unrelated.  But it
should be possible to sort the coordinates to get an top-to-bottom,
left-to-write reading order.  Maybe even with some sensitivity to
BiDi.

I've certainly seen that use case mentioned by others.

> I imagine to do this you'd need DOM-like access to the master slide /
> layout / style, and could then us SAX-like single pass for the
> "normal" slides.
>

Well, you could stream one slide at a time, but we'd need to be able
to store the complete text contents of each individual slide to do the
coordinate sort.  But that is not so bad.  Presentations tend to be
outrageously large based on large images (high color depth, high dpi)
rather than large amounts of text.

> TIKA-735 is another issue with the the current ODF parser, whereby the text
> from embedded documents is always placed at the end of the text from
> the original document, rather than being inlined at the point where
> the embedding occurred.  Seems like a SAX like API would work fine
> here, ie, we should simply recurse into the embedded doc when we
> encounter it.
>

Right.

> Mike McCandless
>
> http://blog.mikemccandless.com
>

Mime
View raw message