corinthia-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis E. Hamilton (JIRA)" <>
Subject [jira] [Created] (COR-31) Identification of Document Format Tool Progressions: Access, Creation, Testing, Assessment, Validation, Forensics
Date Sat, 17 Jan 2015 01:17:34 GMT
Dennis E. Hamilton created COR-31:

             Summary: Identification of Document Format Tool Progressions: Access, Creation,
Testing, Assessment, Validation, Forensics
                 Key: COR-31
             Project: Corinthia
          Issue Type: Task
            Reporter: Dennis E. Hamilton

There are many needs, and opportunities, for command-line and library-level tools that support
the development of processors for different document formats.  

Many small tools can be developed as part of the application and verification of what will
be larger solutions with regard to particular formats.  

This task is for identification of which such tools will be defined as work-product and deliverables
for Corinthia, even in an initial provisional list.  Having an identified structure points
for defined deliverables should aid in having different aspects of Corinthia available for
development and testing by many hands and eyes.


There are different levels of tools, and the layers provide fixtures for exercising lower
layers of code and also composing them into layers above.

To be concrete, here is a sketch of the levels of tooling that can be byproducts and aids
in the confirmation of correct handling of a document format.

There are two "raw" formats that are handled in building document files of interest to us:
text files and Zip packages (or other carriers of composite structures, such as MIME multi-part,
tar files, Microsoft DocFiles, etc.).  

There are flat file formats atop text-file formats.  Examples are Microsoft RTF, XML, and
HTML.  These are accompanied by character-set encoding variations that must be dealt with.
 There are also cases of linking that arise in these formats.

RTF is a document format.  XML carries document formats such as the single-file ODF format,
the single-file XML formats defined for Microsoft Office, etc.  There are already HTML-format
usages that provide for fidelity preservation in round trip between HTML and Microsoft Office
formats.  There may be something similar that has lived in  These are very
handy formats for creation of simple test documents that exercise the respective document
models.  They also provide experience with the document formats and efforts to abstract the
document that is represented in those formats.

Zip usage as carriers raises its own needs for well-defined tools, both for use in the inspection
of document files but also the validation and forensic analysis of the Zip usage for ODF,
OOXML, and other formats, such as ePub.  Now we're dealing with composite document files with
multiple parts using flat formats, such as HTML and XML, and other formats, including binary
formats not mentioned as part of this progressive layering.  There are now more elaborate
structures to abstract from the parts of the Zip package and the cross-references among them.

These are all tooling opportunities and they support the testing and confirmation of the development
of the document-processing functions that Corinthia makes available.

The richness of this can be illustrated by the need for forensic and validation tools and
how they may become interdependent.

Consider the simple verification of a Zip file.  There are two levels of verification that

First there is of the fundamental invariant structure that a Zip archive must possess.  In
practical use, it is desirable to rapidly abstract the presence of a correct Zip and its components.
 It is desirable to be able to produce or update one efficiently.   One wants a fail-safe
and resilient response when an unacceptable Zip is encountered.

At the same time, one wants a way to assess and inspect a Zip that is well-formed or is considered
defective.  A separate tool would be handier for that, but needed to support document processing
by providing inspection and reporting of how the Zip is unacceptable.  That's more involved
and not something one wants to endure just to get going working with a document.  At the same
time, there is a good case for some reused common code as well, and these kinds of tools aid
in the confirmation of that code too.

Suppose a Zip is concluded to be damaged.  Another level is goes beyond detection of damage
to determination of how much of the Zip can be recovered and what to do with the areas of
damage.  This is about rescuing documents.  Yet another opportunity.  Yet another elaborate
use that can involve some shared underlying code.

We're now at the second level and that intersects with the use of a Zip as a particular kind
of document container.  A zip may be well-formed, but there are additional limitations and
functions that go into recognizing the Zip usage as a carrier of a particular document format.
 It can even be a generic carrier format, such as the Open Packaging Conventions (OPC) used
for carrying OOXML, XPS, and other artifacts, and the OpenDocument 1.2 Package used for carrying

There need to be analysis and inspection tools at this second level of generic Zip usage.
 This also has a cross-over value in the forensic problem of recovering what is recoverable
in a damaged Zip archive.  When it is known what additional structure is expected to be present,
this can inform the identification of breakage and determination of loss.

It's not all one-sided.  What appears to be a well-formed Zip package for a given document
format can still expose damage in the recording or compression (oh yes, compression and decompression)
of any of its parts.

This sketch is still at the plumbing level.  The abstraction of document features is yet to
happen.  That's raising up another level.

This is all just to point out how many opportunities for tools and supporting libraries there
are. The tools are important for bootstrapping up the levels of Corinthia and for being able
to check our own work, to devise tests and demonstrations, and to provide forensic support
in the face of problems that may arise in the software or simply in circumstances that arise
for users.

The idea behind this task and its subtasks is to see what could be identified as point deliverables,
even if fundamentally for our own work process, so that they become definable and something
to work on, to be available in higher levels of operation, etc.

This message was sent by Atlassian JIRA

View raw message