Return-Path: Delivered-To: apmail-ws-axis-dev-archive@www.apache.org Received: (qmail 63895 invoked from network); 18 Jul 2007 05:16:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 18 Jul 2007 05:16:08 -0000 Received: (qmail 16876 invoked by uid 500); 18 Jul 2007 05:15:56 -0000 Delivered-To: apmail-ws-axis-dev-archive@ws.apache.org Received: (qmail 16829 invoked by uid 500); 18 Jul 2007 05:15:56 -0000 Mailing-List: contact axis-dev-help@ws.apache.org; run by ezmlm Precedence: bulk Reply-To: axis-dev@ws.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list axis-dev@ws.apache.org Received: (qmail 16815 invoked by uid 99); 18 Jul 2007 05:15:56 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2007 22:15:56 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of amilasuriarachchi@gmail.com designates 209.85.134.187 as permitted sender) Received: from [209.85.134.187] (HELO mu-out-0910.google.com) (209.85.134.187) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2007 22:15:53 -0700 Received: by mu-out-0910.google.com with SMTP id w1so84137mue for ; Tue, 17 Jul 2007 22:15:31 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=GfXVdBcwggZQKpMpTRM9c/sEPquqk7OwGL+p0IunMwoEXxGRoGShNjvFYnFVAsKUE78ppbPd7qI0gJ8n4RH92A5K73/0o+HFD/kELQZEaI4ExH9FWlm2KJy/RfeJaoCLdtVvaVEeQCb78xrSyNUqF2cTGUZgGZfMU4lPrfTZhxE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=s6z84L6goJ/3Cm4ywI1s0CWu63mVbWTu+bI+N5h/m/8EpGOe8LsK+biLZfml2o8zX4zO4C28c/FeunsO+s46t8xXX4gY0Q4ozFCYbzNz1yb0moKvjh5B3zAK1Ac8CnS6WEqIvHBubldE4N4B4V5xTgUyPZul4lN36+LbpoGc42c= Received: by 10.82.100.1 with SMTP id x1mr1443695bub.1184735730683; Tue, 17 Jul 2007 22:15:30 -0700 (PDT) Received: by 10.82.172.13 with HTTP; Tue, 17 Jul 2007 22:15:30 -0700 (PDT) Message-ID: <60708f4b0707172215w53c6932dr8b52b460b2775fc2@mail.gmail.com> Date: Wed, 18 Jul 2007 10:45:30 +0530 From: "Amila Suriarachchi" To: axis-dev@ws.apache.org Subject: Re: Proposal for fixing out-of-memory error JIRA axis2-2968 In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_65281_23538746.1184735730651" References: <60708f4b0707162125v1b8be3e6ka3a36027069a66f4@mail.gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_65281_23538746.1184735730651 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline On 7/17/07, Ann Robinson wrote: > > The Out-Of-Memory (OOM) error happens on a server with a large number of > applications. At deployment time, when the applications are loaded, the > AxisService objects are created. For those applications where the loading > ends up invoking the WSDL11ToAxisServiceBuilder, the wsdl is read in and > saved as a Parameter on the AxisService object. So, the OOM happens before > requests start flowing. > > I tried a solution where the WSDLDefinition object is saved via > serialization, so that it could be re-loaded when needed after deployment > time. The javax.wsdl.Definition interface is supposed to implement the > java.io.Serializable interface. However, the WSDLDefinition object fails > to serialize. I speculate that the WSDLDefinition implementation has some > internal tables or structures that don't support the default java > serialization. > > I tried another solution with a wrapper on the WSDLDefinition object. The > wrapper, which is transparent to the user of the WSDLDefinition object, > releases the heavyweight portions of the WSDLDefinition object. If a > component accesses the WSDLDefinition via the Parameter from the AxisService > object, the wrapper can reload the WSDLDefinition. This solution resolved > the OOM problem - the server was able to load and start the applications > successfully, and handle requests. > Did you tried the ?wsdl for wsdls having 1. imported wsdl in it 2. imported and included schemas on it 3. inline schemas on it with useOriginalWSDL parameter set to true? And wsdl2java generated code for above senarios? I will create a patch with this solution so that it can be reviewed. Ok. Please send a note to this thread as well. > Ann > > > On 07/16/2007 11:25 PM, *"Amila Suriarachchi" >* wrote: > > > On 7/16/07, *Sanjiva Weerawarana* <*sanjiva@opensource.lk*> > wrote: > > This is used for when useOriginalWSDL is true right? > > yes. > > > > I think the best thing is option (a) and have some policy for > taking it out of the server > > and re-loading it if it hasn't been hit for a while. > > > > I'm very much against option (c)- that's overkill for this problem > IMO. > > > > In any case, why is it leading to an OOM error? The WSDL should be > read > > once and then hung onto ... so unless the number of services is > growing > > arbitrarily large there's really no reason for the memory usage to > grow. > > I'm not convinced this is the reason for the OOM (unless for some > reason > > we keep creating new Definitions and never release the old one). > > > yes. this should not cause any OOM errors unless we load wsdl definition > object per request. (then that is clearly an Axis2 bug). Ann are you getting > the error at request processing time or deployment time? > > if it is at request processing time then we should creating wsdl > definition objects per request and that should be an axis2 bug. What we > supposed to do is to keep the original wsdl definition object and serialize > it when user requested. But Here I think we can check for useOriginalWsdl > parameter before storing that in memory to improve performance. > > if it is at deployment time none of above solutions would not solve the > problem. Since we have to create the wsdlDefinition object at least once to > create the axis object structure. > > thanks, > Amila. > > >> On 15/07/07, Ann Robinson <*robinsona@us.ibm.com*> > wrote: > >> > > >> > > >> > Hi, all, > >> > I've been investigating an out-of-memory error that happens > on the server > >> > in some environments. In analyzing the java heap dumps, one > of the biggest > >> > consumers of memory is with the wsdl4j WSDLDefinition objects. > The heap > >> > dump indicates that the WSDLDefinition uses the xerces dom for > underlying > >> > support, particularly for schemas. This makes the > WSDLDefinition very > >> > heavy-weight. > >> > > >> > The heap dump also shows that the WSDLDefinition objects of > concern are the > >> > ones being saved in the AxisService's ParameterInclude list. > Comments in > >> > the code (WSDL11ToAxisServiceBuilder) indicate that this is > done so that, if > >> > some component needs to utilize the WSDLDefinition, the > component can access > >> > it via a Parameter in the AxisService. > >> > > >> > Is it possible to reduce the utilization of the > WSDLDefinition? > >> > Some ideas are: > >> > (a) releasing it when it is no longer needed > >> > - this might not be possible to determine > >> > > >> > (b) putting a wrapper on the WSDLDefinition object > >> > - so that the WSDLDefinition,or a portion of the > WSDLDefinition, > >> > can be released > >> > - but if the WSDLDefinition is accessed after it was > released > >> > the wrapper can reload the WSDLDefinition > transparently to > >> > user > >> > > >> > (c) create a layer for caching wsdl-related information > >> > - this would allow for releasing memory based on > some algorithm > >> > and/or interface that could indicate what's no > longer needed > >> > > >> > Ann > > -- Amila Suriarachchi, WSO2 Inc. ------=_Part_65281_23538746.1184735730651 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline

On 7/17/07, Ann Robinson <robinsona@us.ibm.com> wrote:

The Out-Of-Memory (OOM) error happens on a server with a large number of applications.  At deployment time, when the applications are loaded, the AxisService objects are created. For those applications where the loading ends up invoking the WSDL11ToAxisServiceBuilder, the wsdl is read in and saved as a Parameter on the AxisService object.  So, the OOM happens before requests start flowing.

I tried a solution where the WSDLDefinition object is saved via serialization, so that it could be re-loaded when needed after deployment time. The javax.wsdl.Definition interface is supposed to implement the java.io.Serializable interface. However, the WSDLDefinition object fails to serialize. I speculate that the WSDLDefinition implementation has some internal tables or structures that don't support the default java serialization.

I tried another solution with a wrapper on the WSDLDefinition object.  The wrapper, which is transparent to the user of the WSDLDefinition object, releases the heavyweight portions of the WSDLDefinition object.  If a component accesses the WSDLDefinition via the Parameter from the AxisService object, the wrapper can reload the WSDLDefinition.  This solution resolved the OOM problem - the server was able to load and start the applications successfully, and handle requests.

Did you tried the ?wsdl for wsdls having
1. imported wsdl in it
2. imported and included schemas on it
3. inline schemas on it
with useOriginalWSDL parameter set to true?

And wsdl2java generated code for above senarios?

 
I will create a patch with this solution so that it can be reviewed.

Ok. Please send a note to this thread as well.


Ann


On 07/16/2007 11:25 PM, "Amila Suriarachchi" <amilasuriarachchi@gmail.com > wrote:

 > On 7/16/07, Sanjiva Weerawarana <sanjiva@opensource.lk > wrote:
 > This is used for when useOriginalWSDL is true right?

yes.


    > I think the best thing is option (a) and have some policy for taking it out of the server
    > and re-loading it if it hasn't been hit for a while.
    >
    > I'm very much against option (c)- that's overkill for this problem IMO.
    >
    > In any case, why is it leading to an OOM error? The WSDL should be read
    > once and then hung onto ... so unless the number of services is growing
    > arbitrarily large there's really no reason for the memory usage to grow.
    > I'm not convinced this is the reason for the OOM (unless for some reason
    > we keep creating new Definitions and never release the old one).
 
yes.  this should not cause any OOM errors unless we load wsdl definition object per request. (then that is clearly an Axis2 bug). Ann are you getting the error at request processing time or deployment time?

if it is at request processing time then we should creating wsdl definition objects per request and that should be an axis2 bug. What we supposed to do is to keep the original wsdl definition object and serialize it when user requested. But Here I think we can check for useOriginalWsdl parameter before storing that in memory to improve performance.

if it is at deployment time none of above solutions would not solve the problem. Since we have to create the wsdlDefinition object at least once to create the axis object structure.

thanks,
Amila.

    >>  On 15/07/07, Ann Robinson <robinsona@us.ibm.com > wrote:
    >>  >
    >>  >
    >>  > Hi, all,
    >>  >  I've been investigating an out-of-memory error that happens on the server
    >>  > in some environments.  In analyzing the java heap dumps, one of the biggest
    >>  > consumers of memory is with the wsdl4j WSDLDefinition objects.  The heap
    >>  > dump indicates that the WSDLDefinition uses the xerces dom for underlying
    >>  > support, particularly for schemas.  This makes the WSDLDefinition very
    >>  > heavy-weight.
    >>  >
    >>  >  The heap dump also shows that the WSDLDefinition objects of concern are the
    >>  > ones being saved in the AxisService's ParameterInclude list.  Comments in
    >>  > the code (WSDL11ToAxisServiceBuilder) indicate that this is done so that, if
    >>  > some component needs to utilize the WSDLDefinition, the component can access
    >>  > it via a Parameter in the AxisService.
    >>  >
    >>  >  Is it possible to reduce the utilization of the WSDLDefinition?
    >>  >  Some ideas are:
    >>  >  (a) releasing it when it is no longer needed
    >>  >           - this might not be possible to determine
    >>  >
    >>  >  (b) putting a wrapper on the WSDLDefinition object
    >>  >           - so that the WSDLDefinition,or a portion of the WSDLDefinition,
    >>  >             can be released
    >>  >           - but if the WSDLDefinition is accessed after it was released
    >>  >             the wrapper can reload the WSDLDefinition transparently to
    >>  >             user
    >>  >
    >>  >  (c) create a layer for caching wsdl-related information
    >>  >           - this would allow for releasing memory based on some algorithm
    >>  >             and/or interface that could indicate what's no longer needed
    >>  >
    >>  >  Ann




--
Amila Suriarachchi,
WSO2 Inc. ------=_Part_65281_23538746.1184735730651--