uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Muon Le" <muon...@temis.com>
Subject RE: UIMA-2.3 : CPE timeout problem with remote processors.
Date Thu, 08 Apr 2010 08:55:30 GMT
Thank you Jaroslaw for you quick anwsere. I tried your recommandation and added <timeout>10000</timeout>
in the client description of Vinci.
However the problem of hang of CPE is still there.

Here are log messages of client side and remote site and description files.
We can see the timeout value of CPE (30000ms) is taken account and the error of timeout appears
after 30000ms.

I hope that some messages send by org.apache.vinci.transport.BaseServer about Pooled information
can help to understand the situation.

Thanks for your helps.
Muon.

PS.

In client site, the CPE logs these messages :

DEBUG - 2010-04-08 10:21:43,453 - Sending Process Request to Service. (Thread Name: [Procesing
Pipeline#2 Thread]::) Service Host: ??? Service Port: 10148 Timeout: 30000 - [MDC:{{module,Workflow}{host,???}{submodule,???}{workflow,Bug5092}}][THREAD:[Procesing
Pipeline#2 Thread]::][LOGGER:org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP][NDC:null]
WARN  - 2010-04-08 10:22:13,843 - A service error occurred when processing a document. (Thread
Name: [Procesing Pipeline#4 Thread]::) Service Host: ??? Service Port: 10158 - [MDC:{{module,Workflow}{host,???}{submodule,???}{workflow,Bug5092}}][THREAD:[Procesing
Pipeline#4 Thread]::][LOGGER:org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP][NDC:null]
WARN  - 2010-04-08 10:22:13,843 - [Procesing Pipeline#4 Thread]:: - [MDC:{{module,Workflow}{host,???}{submodule,???}{workflow,Bug5092}}][THREAD:[Procesing
Pipeline#4 Thread]::][LOGGER:org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP][NDC:null]
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	at org.apache.vinci.transport.XTalkTransporter.fromStream(XTalkTransporter.java:61)
	at org.apache.vinci.transport.Frame.fromStream(Frame.java:150)
	at org.apache.vinci.transport.BaseClient.sendAndReceiveWork(BaseClient.java:650)
	at org.apache.vinci.transport.BaseClient.sendAndReceiveWork(BaseClient.java:666)
	at org.apache.vinci.transport.BaseClient.sendAndReceive(BaseClient.java:383)
	at org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP.sendAndReceive(VinciTAP.java:453)
	at org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP.analyze(VinciTAP.java:927)
	at org.apache.uima.collection.impl.cpm.container.NetworkCasProcessorImpl.process(NetworkCasProcessorImpl.java:198)
	at org.apache.uima.collection.impl.cpm.engine.ProcessingUnit.processNext(ProcessingUnit.java:826)
	at org.apache.uima.collection.impl.cpm.engine.ProcessingUnit.run(ProcessingUnit.java:577)

In remote processor site, UIMA logs these messages :
[4/8/10 10:21 AM] Pooledcount: 1 busyCount: 0
[4/8/10 10:21 AM] Pooledcount: 1 busyCount: 0
[4/8/10 10:21 AM] Pooledcount: 1 busyCount: 0
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 1
[4/8/10 10:22 AM] Creating a thread for pool of current size 1
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 1
[4/8/10 10:22 AM] Creating a thread for pool of current size 1
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 1
[4/8/10 10:22 AM] Creating a thread for pool of current size 1
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 1
[4/8/10 10:22 AM] Creating a thread for pool of current size 1
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 2
[4/8/10 10:22 AM] Creating a thread for pool of current size 2
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 2
[4/8/10 10:22 AM] Creating a thread for pool of current size 2
[4/8/10 10:22 AM] Pooledcount: 0 busyCount: 2
[4/8/10 10:22 AM] Creating a thread for pool of current size 2
[4/8/10 10:23 AM] Pooledcount: 0 busyCount: 3
[4/8/10 10:23 AM] Creating a thread for pool of current size 3
[4/8/10 10:23 AM] Pooledcount: 0 busyCount: 3
[4/8/10 10:23 AM] Creating a thread for pool of current size 3

CPE description :

<?xml version="1.0" encoding="UTF-8"?>
<cpeDescription xmlns="http://uima.apache.org/resourceSpecifier">
    <collectionReader>
        <collectionIterator>
            <descriptor>
                <include href="/F:/.../client/descriptors/collection_readers/XCasCollectionReader.xml"/>
            </descriptor>
            <configurationParameterSettings>
                <nameValuePair>
                    <name>InputDirectory</name>
                    <value>
                        <string>g:/.../bugs/5092/data/xcas</string>
                    </value>
                </nameValuePair>
            </configurationParameterSettings>
        </collectionIterator>
    </collectionReader>
    <casProcessors casPoolSize="5" processingUnitThreadCount="4">
        <casProcessor deployment="remote" name="SleepBSHOutCas">
            <descriptor>
                <include href="c:\...\AnnotClient_Default_SleepBSHOutCas.xml"/>
            </descriptor>
            <deploymentParameters>
                <parameter name="vnsHost" type="string" value="???"/>
                <parameter name="vnsPort" type="string" value="9000"/>
                <parameter name="service-access" type="string" value="exclusive"/>
            </deploymentParameters>
            <errorHandling>
                <errorRateThreshold action="continue" value="1/1000"/>
                <maxConsecutiveRestarts action="continue" value="0"/>
                <timeout max="30000" default="-1"/>
            </errorHandling>
            <checkpoint batch="1" time="1000ms"/>
        </casProcessor>
        <casProcessor deployment="integrated" name="CasConsumer">
            <descriptor>
                <include href="/F:/.../client/descriptors/cas_consumers/XCasWriterCasConsumer.xml"/>
            </descriptor>
            <deploymentParameters/>
            <errorHandling>
                <errorRateThreshold action="terminate" value="100/1000"/>
                <maxConsecutiveRestarts action="terminate" value="30"/>
                <timeout max="100000" default="-1"/>
            </errorHandling>
            <checkpoint batch="1" time="1000ms"/>
            <configurationParameterSettings>
                <nameValuePair>
                    <name>OutputDirectory</name>
                    <value>
                        <string>c:/temp/XCasDirectory</string>
                    </value>
                </nameValuePair>
            </configurationParameterSettings>
        </casProcessor>
    </casProcessors>
    <cpeConfig>
        <numToProcess>-1</numToProcess>
        <deployAs>immediate</deployAs>
        <checkpoint batch="0" time="300000ms"/>
        <timerImpl/>
    </cpeConfig>
</cpeDescription>


Client description of Vinci service :

<?xml version="1.0" encoding="UTF-8"?><uriSpecifier xmlns="http://uima.apache.org/resourceSpecifier">
    <resourceType>AnalysisEngine</resourceType>
    <uri>com.temis.Default.SleepBSHOutCas</uri>
    <protocol>Vinci</protocol>
    <timeout>10000</timeout>
</uriSpecifier>

Description of the service deployment :
<?xml version="1.0" encoding="UTF-8"?>
<deployment name="Vinci SleepBSHOutCas Service">
  <service name="com.temis.Default.SleepBSHOutCas" provider="vinci">
    <parameter name="serializerClassName" value="org.apache.uima.adapter.vinci.VinciXCASSerializer_NoDocText"/>
    <parameter name="resourceSpecifierPath" value="F:\...\SleepBSHOutCas\desc\analysis_engine\SleepBSHOutCas.xml"/>
    <parameter name="numInstances" value="1"/>
    <parameter name="serverSocketTimeout" value="1800000"/>
  </service>
</deployment> 	


//////////////////////////////////////////// END ///////////////////////////////////////////















-----Message d'origine-----
De : Jaroslaw Cwiklik [mailto:uimaee@gmail.com] 
Envoyé : mercredi 7 avril 2010 16:19
À : uima-user@incubator.apache.org
Objet : Re: UIMA-2.3 : CPE timeout problem with remote processors.

Muon, can you change your client descriptor to set a timeout to say 10000
(ms)? Your descriptor is not setting
this at all and relies on a default timeout value which I believe is quite
short. Just add this to your descriptor:
<timeout>10000</timeout>

This is your descriptor:

 <?xml version="1.0" encoding="UTF-8" ?>
- <uriSpecifier xmlns="http://uima.apache.org/resourceSpecifier">
 <resourceType>AnalysisEngine</resourceType>
 <uri>com.temis.Default.SleepBSH</uri>
 <protocol>Vinci</protocol>
 </uriSpecifier>


The exception you've provided in a previous post indicates a socket timeout
which should be fixed by the suggestion above.
Also you may want to crank up debug level to FINEST to see what the timeout
is on a send. The timeout is expressed in ms.

Yes, here is a sample of error message in logs:
WARN  - 2010-03-29 16:09:44,312 - [Procesing Pipeline#4 Thread]:: -
[MDC:{{docid,
http://www.ardenno.com/pressreleases/news_04sept07.php--Cnt6183}{module,Workflow}{host,racine}{submodule,racine}{workflow,Bug5092}}][THREAD:[Procesing<http://www.ardenno.com/pressreleases/news_04sept07.php--Cnt6183%7D%7Bmodule,Workflow%7D%7Bhost,racine%7D%7Bsubmodule,racine%7D%7Bworkflow,Bug5092%7D%7D][THREAD:[Procesing>
Pipeline#4
Thread]::][LOGGER:org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP][NDC:null]
java.net.SocketTimeoutException: Read timed out
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(Unknown Source)
       at java.io.BufferedInputStream.fill(Unknown Source)
       at java.io.BufferedInputStream.read(Unknown Source)
       at
org.apache.vinci.transport.XTalkTransporter.fromStream(XTalkTransporter.java:61)
       at org.apache.vinci.transport.Frame.fromStream(Frame.java:150)
       at
org.apache.vinci.transport.BaseClient.sendAndReceiveWork(BaseClient.java:650)
       at
org.apache.vinci.transport.BaseClient.sendAndReceiveWork(BaseClient.java:666)
       at
org.apache.vinci.transport.BaseClient.sendAndReceive(BaseClient.java:383)
       at
org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP.sendAndReceive(VinciTAP.java:453)
       at
org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP.analyze(VinciTAP.java:927)
       at
org.apache.uima.collection.impl.cpm.container.NetworkCasProcessorImpl.process(NetworkCasProcessorImpl.java:198)
       at
org.apache.uima.collection.impl.cpm.engine.ProcessingUnit.processNext(ProcessingUnit.java:826)
       at
org.apache.uima.collection.impl.cpm.engine.ProcessingUnit.run(ProcessingUnit.java:577)

On Mon, Apr 5, 2010 at 6:16 PM, Muon Le <muon.le@temis.com> wrote:

> Joroslaw,
>
> With the same big CAS processed twice, my remote processor receive the CAS
> the firt time, not the second time. In the second time, the CPE hangs.
>
> Thank you for your suggestion, we probably plan to use UIMA-AS, but for the
> moment, we work with UIMA-2.3.
>
> Thanks for your helps.
>
> Muon.
>
> -----Message d'origine-----
> De : Jaroslaw Cwiklik [mailto:uimaee@gmail.com]
> Envoyé : lundi 5 avril 2010 16:37
> À : uima-user@incubator.apache.org
> Objet : Re: UIMA-2.3 : CPE timeout problem with remote processors.
>
> Muon, can you try to send the same big CAS (the first that worked) again to
> see if the CPE handles the timeout correctly? Does your service receive the
> second large CAS? I am trying to figure out if this is a problem with
> serializing content of the second large CAS or something else.
>
> Just a side note, if you are evaluating UIMA remoting capabilities, perhaps
> you should try UIMA AS instead. UIMA AS is a replacement for the CPM. Based
> on Open Source ActiveMQ, UIMA AS is a highly scalable framework for
> deploying remote analytics. It scales to thousands of compute cores. Here
> is
> a link where you'll find additional information:
>
> http://incubator.apache.org/uima/doc-uimaas-what.html
>
> JC
>
> Muon, can you try to dispose previous instance of the CPE and create a new
> one before sending another big CAS?
>
> On Thu, Apr 1, 2010 at 3:05 AM, Muon Le <muon.le@temis.com> wrote:
>
> > I did more tests about bigs XCas files (>100Ko).
> > When I start my CPE with only one big XCas file, the CPE terminates
> > properly with one timout error.
> > Howerver, when I continue to start the same CPE with an other big XCas
> > file, the CPE detects the timeout error and hangs.
> >
> > Thanks,
> > Muon.
> >
> > -----Message d'origine-----
> > De : Jaroslaw Cwiklik [mailto:uimaee@gmail.com]
> > Envoyé : mercredi 31 mars 2010 18:12
> > À : uima-user@incubator.apache.org
> > Objet : Re: UIMA-2.3 : CPE timeout problem with remote processors.
> >
> > Hmm, your  previous description confused me:
> >
> > "More accuratly, with small files, there are errors about timeout, but
> the
> > CPE terminates properly (without handling)."
> >
> > So you are not seeing any problems while processing small files. Its the
> > big
> > files that cause the timeout. Looking at
> > the stack trace it seems like this is a problem in the Vinci
> > Serializer/Deserializer. Not sure if this a problem with
> > sending or receiving. Is your service receiving and processing a large
> CAS
> > (file)? How large is the file?
> >
> > JC
> >
> > On Wed, Mar 31, 2010 at 10:27 AM, Muon Le <muon.le@temis.com> wrote:
> >
> > > Hello,
> > >
> > > No, with small files, there are no problems.
> > >
> > > No. Here is the description file:
> > >   <?xml version="1.0" encoding="UTF-8" ?>
> > > - <uriSpecifier xmlns="http://uima.apache.org/resourceSpecifier">
> > >  <resourceType>AnalysisEngine</resourceType>
> > >  <uri>com.temis.Default.SleepBSH</uri>
> > >  <protocol>Vinci</protocol>
> > >  </uriSpecifier>
> > >
> > > Yes, here is a sample of error message in logs:
> > > WARN  - 2010-03-29 16:09:44,312 - [Procesing Pipeline#4 Thread]:: -
> > > [MDC:{{docid,
> > >
> >
> http://www.ardenno.com/pressreleases/news_04sept07.php--Cnt6183}{module,Workflow}{host,racine}{submodule,racine}{workflow,Bug5092}}][THREAD:[ProcesingPipeline#4
> > >
> >
> Thread]::][LOGGER:org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP][NDC:null]
> > > java.net.SocketTimeoutException: Read timed out
> > >        at java.net.SocketInputStream.socketRead0(Native Method)
> > >        at java.net.SocketInputStream.read(Unknown Source)
> > >        at java.io.BufferedInputStream.fill(Unknown Source)
> > >        at java.io.BufferedInputStream.read(Unknown Source)
> > >        at
> > >
> >
> org.apache.vinci.transport.XTalkTransporter.fromStream(XTalkTransporter.java:61)
> > >        at org.apache.vinci.transport.Frame.fromStream(Frame.java:150)
> > >        at
> > >
> >
> org.apache.vinci.transport.BaseClient.sendAndReceiveWork(BaseClient.java:650)
> > >        at
> > >
> >
> org.apache.vinci.transport.BaseClient.sendAndReceiveWork(BaseClient.java:666)
> > >        at
> > >
> org.apache.vinci.transport.BaseClient.sendAndReceive(BaseClient.java:383)
> > >        at
> > >
> >
> org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP.sendAndReceive(VinciTAP.java:453)
> > >        at
> > >
> >
> org.apache.uima.collection.impl.cpm.container.deployer.VinciTAP.analyze(VinciTAP.java:927)
> > >        at
> > >
> >
> org.apache.uima.collection.impl.cpm.container.NetworkCasProcessorImpl.process(NetworkCasProcessorImpl.java:198)
> > >        at
> > >
> >
> org.apache.uima.collection.impl.cpm.engine.ProcessingUnit.processNext(ProcessingUnit.java:826)
> > >        at
> > >
> >
> org.apache.uima.collection.impl.cpm.engine.ProcessingUnit.run(ProcessingUnit.java:577)
> > >
> > > Thanks for your helps.
> > > Muon.
> > >
> > > -----Message d'origine-----
> > > De : Jaroslaw Cwiklik [mailto:uimaee@gmail.com]
> > > Envoyé : mercredi 31 mars 2010 15:39
> > > À : uima-user@incubator.apache.org
> > > Objet : Re: UIMA-2.3 : CPE timeout problem with remote processors.
> > >
> > > Do you also observe a hang with small files?
> > > Does client descriptor c:\temp\AnnotClient_Default_SleepBSH.xml name
> the
> > > same vns host and port that the service is using?
> > > Any exceptions in logs( service or client side)?
> > > JC
> > >
> > > On Tue, Mar 30, 2010 at 9:21 AM, Muon Le <muon.le@temis.com> wrote:
> > >
> > > > Hello,
> > > >
> > > >
> > > >
> > > > I launch a simple CPE (see bellow) that maintains a remote processor
> > > > with a timeout defined at 30000ms.
> > > >
> > > > The CPE hangs when the processor processes bigs files (10 files >
> > 100ko)
> > > > despite the timeout value.
> > > >
> > > >
> > > >
> > > > Do you have any idea of this problem ?
> > > >
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > > Muon.
> > > >
> > > > PS. My CPE description :
> > > >
> > > >
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"?>
> > > >
> > > > <cpeDescription xmlns="http://uima.apache.org/resourceSpecifier">
> > > >
> > > >    <collectionReader>
> > > >
> > > >        <collectionIterator>
> > > >
> > > >            <descriptor>
> > > >
> > > >                <include
> > > >
> > href="/F:/...../descriptors/collection_readers/XCasCollectionReader.xml"
> > > > />
> > > >
> > > >            </descriptor>
> > > >
> > > >            <configurationParameterSettings>
> > > >
> > > >                <nameValuePair>
> > > >
> > > >                    <name>InputDirectory</name>
> > > >
> > > >                    <value>
> > > >
> > > >                        <string>g:/Luxid/bugs/5092/data/xcas</string>
> > > >
> > > >                    </value>
> > > >
> > > >                </nameValuePair>
> > > >
> > > >            </configurationParameterSettings>
> > > >
> > > >        </collectionIterator>
> > > >
> > > >    </collectionReader>
> > > >
> > > >    <casProcessors casPoolSize="5" processingUnitThreadCount="4">
> > > >
> > > >        <casProcessor deployment="remote" name="SleepBSH">
> > > >
> > > >            <descriptor>
> > > >
> > > >                <include
> > > > href="c:\temp\AnnotClient_Default_SleepBSH.xml"/>
> > > >
> > > >            </descriptor>
> > > >
> > > >            <deploymentParameters>
> > > >
> > > >                <parameter name="vnsHost" type="string" value=" ???"/>
> > > >
> > > >                <parameter name="vnsPort" type="string" value="9000"/>
> > > >
> > > >                <parameter name="service-access" type="string"
> > > > value="exclusive"/>
> > > >
> > > >            </deploymentParameters>
> > > >
> > > >            <errorHandling>
> > > >
> > > >                <errorRateThreshold action="continue" value="1/1000"/>
> > > >
> > > >                <maxConsecutiveRestarts action="continue" value="0"/>
> > > >
> > > >                <timeout max="30000" default="-1"/>
> > > >
> > > >            </errorHandling>
> > > >
> > > >            <checkpoint batch="1" time="1000ms"/>
> > > >
> > > >        </casProcessor>
> > > >
> > > >        <casProcessor deployment="integrated" name="CasConsumer">
> > > >
> > > >            <descriptor>
> > > >
> > > >                <include
> > > >
> href="/F:/....../descriptors/cas_consumers/XCasWriterCasConsumer.xml"/>
> > > >
> > > >            </descriptor>
> > > >
> > > >            <deploymentParameters/>
> > > >
> > > >            <errorHandling>
> > > >
> > > >                <errorRateThreshold action="terminate"
> > > > value="100/1000"/>
> > > >
> > > >                <maxConsecutiveRestarts action="terminate"
> value="30"/>
> > > >
> > > >                <timeout max="100000" default="-1"/>
> > > >
> > > >            </errorHandling>
> > > >
> > > >            <checkpoint batch="1" time="1000ms"/>
> > > >
> > > >            <configurationParameterSettings>
> > > >
> > > >                <nameValuePair>
> > > >
> > > >                    <name>OutputDirectory</name>
> > > >
> > > >                    <value>
> > > >
> > > >                        <string>c:/temp/XCasDirectory</string>
> > > >
> > > >                    </value>
> > > >
> > > >                </nameValuePair>
> > > >
> > > >            </configurationParameterSettings>
> > > >
> > > >        </casProcessor>
> > > >
> > > >    </casProcessors>
> > > >
> > > >    <cpeConfig>
> > > >
> > > >        <numToProcess>-1</numToProcess>
> > > >
> > > >        <deployAs>immediate</deployAs>
> > > >
> > > >        <checkpoint batch="0" time="300000ms"/>
> > > >
> > > >        <timerImpl/>
> > > >
> > > >    </cpeConfig>
> > > >
> > > > </cpeDescription>
> > > >
> > > >
> > >
> >
>

Mime
View raw message