corinthia-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis E. Hamilton (JIRA)" <>
Subject [jira] [Commented] (COR-31) Identification of Document Format Tool Progressions: Access, Creation, Testing, Assessment, Validation, Forensics
Date Sat, 17 Jan 2015 17:13:34 GMT


Dennis E. Hamilton commented on COR-31:

Verification and assessment of XML becomes an interesting factor as well.  I just ran into
a performance-related and even energy-cost argument on the ODF TC comment list:

For OOXML and, in particular, ODF, the assessment of XML validity, possible "correction",
and dealing with allowed extensions are an interesting topic.  The extension mechanism (MCT)
of OOXML is already in use.  There are extended uses in ODF as well, though not in such a
disciplined manner.

More forensic tools, more analysis tools opportunity.  Some might be gathered via existing
XML processing functions, and some might need to be adjusted for application in the context
of Corinthia.  For example, the manner in which unknown or unsupported features are encapsulated
in abstracted forms and translations, for potential recovery on further conversion seems related
to some of the DocFormat ideas.

> Identification of Document Format Tool Progressions: Access, Creation, Testing, Assessment,
Validation, Forensics
> -----------------------------------------------------------------------------------------------------------------
>                 Key: COR-31
>                 URL:
>             Project: Corinthia
>          Issue Type: Task
>            Reporter: Dennis E. Hamilton
>              Labels: document-forensics, document-format, document-standards, document-validation,
documents, file-structure, test-suite
> There are many needs, and opportunities, for command-line and library-level tools that
support the development of processors for different document formats.  
> Many small tools can be developed as part of the application and verification of what
will be larger solutions with regard to particular formats.  
> This task is for identification of which such tools will be defined as work-product and
deliverables for Corinthia, even in an initial provisional list.  Having an identified structure
points for defined deliverables should aid in having different aspects of Corinthia available
for development and testing by many hands and eyes.
> There are different levels of tools, and the layers provide fixtures for exercising lower
layers of code and also composing them into layers above.
> To be concrete, here is a sketch of the levels of tooling that can be byproducts and
aids in the confirmation of correct handling of a document format.
> There are two "raw" formats that are handled in building document files of interest to
us: text files and Zip packages (or other carriers of composite structures, such as MIME multi-part,
tar files, Microsoft DocFiles, etc.).  
> There are flat file formats atop text-file formats.  Examples are Microsoft RTF, XML,
and HTML.  These are accompanied by character-set encoding variations that must be dealt with.
 There are also cases of linking that arise in these formats.
> RTF is a document format.  XML carries document formats such as the single-file ODF format,
the single-file XML formats defined for Microsoft Office, etc.  There are already HTML-format
usages that provide for fidelity preservation in round trip between HTML and Microsoft Office
formats.  There may be something similar that has lived in  These are very
handy formats for creation of simple test documents that exercise the respective document
models.  They also provide experience with the document formats and efforts to abstract the
document that is represented in those formats.
> Zip usage as carriers raises its own needs for well-defined tools, both for use in the
inspection of document files but also the validation and forensic analysis of the Zip usage
for ODF, OOXML, and other formats, such as ePub.  Now we're dealing with composite document
files with multiple parts using flat formats, such as HTML and XML, and other formats, including
binary formats not mentioned as part of this progressive layering.  There are now more elaborate
structures to abstract from the parts of the Zip package and the cross-references among them.
> These are all tooling opportunities and they support the testing and confirmation of
the development of the document-processing functions that Corinthia makes available.
> The richness of this can be illustrated by the need for forensic and validation tools
and how they may become interdependent.
> Consider the simple verification of a Zip file.  There are two levels of verification
that matter.  
> First there is of the fundamental invariant structure that a Zip archive must possess.
 In practical use, it is desirable to rapidly abstract the presence of a correct Zip and its
components.  It is desirable to be able to produce or update one efficiently.   One wants
a fail-safe and resilient response when an unacceptable Zip is encountered.
> At the same time, one wants a way to assess and inspect a Zip that is well-formed or
is considered defective.  A separate tool would be handier for that, but needed to support
document processing by providing inspection and reporting of how the Zip is unacceptable.
 That's more involved and not something one wants to endure just to get going working with
a document.  At the same time, there is a good case for some reused common code as well, and
these kinds of tools aid in the confirmation of that code too.
> Suppose a Zip is concluded to be damaged.  Another level is goes beyond detection of
damage to determination of how much of the Zip can be recovered and what to do with the areas
of damage.  This is about rescuing documents.  Yet another opportunity.  Yet another elaborate
use that can involve some shared underlying code.
> We're now at the second level and that intersects with the use of a Zip as a particular
kind of document container.  A zip may be well-formed, but there are additional limitations
and functions that go into recognizing the Zip usage as a carrier of a particular document
format.  It can even be a generic carrier format, such as the Open Packaging Conventions (OPC)
used for carrying OOXML, XPS, and other artifacts, and the OpenDocument 1.2 Package used for
carrying ODF.
> There need to be analysis and inspection tools at this second level of generic Zip usage.
 This also has a cross-over value in the forensic problem of recovering what is recoverable
in a damaged Zip archive.  When it is known what additional structure is expected to be present,
this can inform the identification of breakage and determination of loss.
> It's not all one-sided.  What appears to be a well-formed Zip package for a given document
format can still expose damage in the recording or compression (oh yes, compression and decompression)
of any of its parts.
> This sketch is still at the plumbing level.  The abstraction of document features is
yet to happen.  That's raising up another level.
> This is all just to point out how many opportunities for tools and supporting libraries
there are. The tools are important for bootstrapping up the levels of Corinthia and for being
able to check our own work, to devise tests and demonstrations, and to provide forensic support
in the face of problems that may arise in the software or simply in circumstances that arise
for users.
> The idea behind this task and its subtasks is to see what could be identified as point
deliverables, even if fundamentally for our own work process, so that they become definable
and something to work on, to be available in higher levels of operation, etc.

This message was sent by Atlassian JIRA

View raw message