ode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nowakowski, Mateusz" <Mateusz.Nowakow...@sabre-holdings.com>
Subject The execution queue is backed up... Forcing ContextException + MEMORY LEAK ??
Date Tue, 04 Dec 2007 11:34:38 GMT
Yes, I use 120 threads for ode and  it is probably maximum considering
that every service mix components need to have similar own thread pool.


I launched longer stress test for 100 clients and for 50 clients. For
100 clients application crashed after 3h.

For 50 clients application survives 10h stress test, but gc log indicate
slow but memory leak. I investigate it, it can be service mix...

but I see the same exception all the time:

15:02:43,927 | ERROR | pool-10-thread-114 | SimpleScheduler          |
duler.simple.SimpleScheduler$4  353 | Error while executing transaction
org.apache.ode.bpel.iapi.ContextException: The execution queue is backed
up... Forcing ContextException
        at
org.apache.ode.scheduler.simple.SimpleScheduler.schedulePersistedJob(Sim
pleScheduler.java:227)
        at
org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.
java:290)
        at
org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.
java:373)
        at
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
.java:341)
        at
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
.java:340)
        at
org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleSc
heduler.java:179)
        at
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
ava:339)
        at
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
ava:336)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:650)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:675)
        at java.lang.Thread.run(Thread.java:595) 


It is impossible to fill ode thread poll due to it is set to 120
threads, and the maximum parallel requests are 50.

I dig further and see that SimpleScheduler uses hardcoded size of
concurrent task which is 200. Still maximum concurrent requests is 50.

What is done with the request when this exception is thrown ??
How to get rid of this ??

Please help.

-- 
Regards
Mateusz Nowakowski

-----Original Message-----
From: matthieu.riou@gmail.com [mailto:matthieu.riou@gmail.com] On Behalf
Of Matthieu Riou
Sent: Thursday, November 29, 2007 5:21 PM
To: dev@ode.apache.org
Subject: Re: The execution queue is backed up... Forcing
ContextException

Do you saturate your CPU (about 85-90%)? If not, more threads should
bring
you more performance at least for in-memory. For persistent processes,
it's
the database optimization game...

Cheers,
Matthieu

On Nov 29, 2007 8:01 AM, Nowakowski, Mateusz <
Mateusz.Nowakowski@sabre-holdings.com> wrote:

> Thanks. It helps a bit. Application doesn't crash anymore but
> performance is decreased than for 50 parallel clients... But it can be
> different reason (not enough RAM or CPU)
>
> I know about this property before but I don't use because I assume
that
> BPEL caches threads and creates new thread when necessary.
> I looked on ODEService.initBpelServer() and saw:
>
>        if (_ode._config.getThreadPoolMaxSize() == 0)
>            _ode._executorService = Executors.newCachedThreadPool();
>        else
>            _ode._executorService =
> Executors.newFixedThreadPool(_ode._config.getThreadPoolMaxSize());
>
>
> --
> Regards
> Mateusz Nowakowski
>
> -----Original Message-----
> From: matthieu.riou@gmail.com [mailto:matthieu.riou@gmail.com] On
Behalf
> Of Matthieu Riou
> Sent: Wednesday, November 28, 2007 5:17 PM
> To: dev@ode.apache.org
> Subject: Re: The execution queue is backed up... Forcing
> ContextException
>
> Hi Mateusz,
>
> I've guessing you're exhausting ODE's thread pool. When the engine
can't
> keep up anymore and messages are coming faster that they can be
> consumed,
> the execution queue gets backed up and instead of accumulating
messages
> we
> throw an exception.
>
> To increase the size of the pool, you can set the following property
in
> ODE's configuration:
>
> ode-jbi.threads.pool.size=100
>
> Let us know how it goes.
>
> Cheers,
> Matthieu
>
> On Nov 28, 2007 6:56 AM, Nowakowski, Mateusz <
> Mateusz.Nowakowski@sabre-holdings.com> wrote:
>
> > I've performing some performance tests using Fuse 3.3.0.2 and Apache
> ODE
> > 1.1.
> >
> > I tested it using 100 parallel clients.
> > I used such thread pool properties:
> > servicemix.corePoolSize    = 32
> > servicemix.maximumPoolSize = 256
> > servicemix.queueSize       = 256
> >
> > I started to get such exception:
> >
> > 05:06:21,891 | ERROR | pool-2-thread-359 | SimpleScheduler
|
> > duler.simple.SimpleScheduler$4  347 | Error while executing
> transaction
> > org.apache.ode.bpel.iapi.ContextException: The execution queue is
> backed
> > up... Forcing ContextException
> >        at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler.schedulePersistedJob(Sim
> > pleScheduler.java:221)
> >        at
> >
>
org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.
> > java:279)
> >        at
> >
>
org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.
> > java:364)
> >        at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> > .java:335)
> >        at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> > .java:334)
> >        at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleSc
> > heduler.java:173)
> >        at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> > ava:333)
> >        at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> > ava:330)
> >        at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> >        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> >        at
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
> > r.java:650)
> >        at
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> > va:675)
> >        at java.lang.Thread.run(Thread.java:595)
> >
> >
> > After some minutes application started going to crash (leak memory
and
> > finally OutOfMemoryError).
> >
> > What is the reason ?
> >
> > When I tested 50 parallel clients everything was ok.
> >
> > --
> > Regards
> > Mateusz Nowakowski
> >
>

Mime
View raw message