Mailing-List: contact user-help@uima.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@uima.apache.org
From: "D. Heinze" <dheinze@gnoetics.com>
To: <user@uima.apache.org>
References: <0a3601d14cdd$d578c440$806a4cc0$@gnoetics.com>
 <881F0CB3-1365-4AE4-BFD5-0229138458E4@apache.org>
In-Reply-To: <881F0CB3-1365-4AE4-BFD5-0229138458E4@apache.org>
Subject: RE: CAS serializationWithCompression
Date: Tue, 12 Jan 2016 07:54:06 -0800
Organization: Gnoetics, Inc.
Message-ID: <0acb01d14d51$77c25810$67470830$@gnoetics.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AQHPnreCMvhREve7kn7+xlX8t6fRDQICSW6NnutPqMA=
Content-Language: en-us

The CAS.size() starts as larger than the serializedWithCompression version,
but eventually the serializedWithCompression version grows to be larger than
the CAS.size().
The overall process is:
* Create a new CAS
* Read in an xml document and store the structure and content in the cas.
* Tokenize and parse the document and store that info in the cas.
* Run a number of lexical engines and ConceptMapper engines on the data and
store that info in the cas
* Produce an xml document with the content of the original input document
marked up with the analysis results and both write that out to a file and
also store it in the cas
* serializeWithCompression to a FileOutputStream
* cas.reset()
* iterate on the next input document
All the work other than creating and cas.reset() is done using the JCas.
Even though the output CASes keep getting larger, they seem to deserialize
just fine and are usable.
Thanks/Dan

-----Original Message-----
From: Richard Eckart de Castilho [mailto:rec@apache.org] 
Sent: Tuesday, January 12, 2016 2:45 AM
To: user@uima.apache.org
Subject: Re: CAS serializationWithCompression

Is the CAS.size() larger than the serialized version or smaller?
What are you actually doing to the CAS? Just serializing/deserializing a
couple of times in a row, or do you actually add feature structures?
The sample code you show doesn't give any hint about where the CAS comes
from and what is being done with it.

-- Richard

> On 12.01.2016, at 03:06, D. Heinze <dheinze@gnoetics.com> wrote:
> 
> I'm having a problem with CAS serializationWithCompression.  I am 
> processing a few million text document on an IBM P8 with 16 physical 
> SMTP 8 cpus, 200GB RAM, Ubuntu 14.04.3 LTS and IBM Java 1.8.
> 
> I run 55 UIMA pipelines concurrently.  I'm using UIMA 2.6.0.
> 
> I use serializeWithCompression to save the final state of the 
> processing on each document to a file for later processing.
> 
> However, the size of the serialized CAS just keeps growing.  The size 
> of the CAS is stable, but the serialized CASes just keep getting 
> bigger. I even went to creating a new CAS for each process instead of 
> using cas.reset().  I have also tried writing the serialized CAS to a 
> byte array output stream first and then to a file, but it is the 
> serializeWithCompression that caused the size problem not writing the
file.
> 
> Here's what the code looks like.  Flushing or not flushing does not 
> make a difference.  Closing or not closing the file output strem does 
> not make a difference (other than leaking memory).  I've also tried 
> doing serializeWithCompression with type filtering.  Wanted to try 
> using a Marker, but cannot see how to do that.  The problem exists 
> regardless of doing 1 or
> 55 pipelines concurrently.
> 
> 
> 
>        File fout = new File(documentPath);
> 
>        fos = new FileOutputStream(fout);
> 
>        
> org.apache.uima.cas.impl.Serialization.serializeWithCompression(
> cas, fos);
> 
>        fos.flush();
> 
>        fos.close();
> 
>        logger.info( "serializedCas size " + cas.size() + " ToFile " + 
> documentPath);
> 
> 
> 
> Suggestions will be appreciated.
> 
> 
> 
> Thanks / Dan
> 
> 
>