axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Molettiere <pie...@axonstudios.net>
Subject Re: AXIS-1423 : Peter's test case for performance
Date Tue, 20 Jul 2004 20:35:47 GMT

Has anyone looked at this bug in xerces?

http://nagoya.apache.org/jira/browse/XERCESJ-237

We're thinking maybe that it might have something to do with the Out of  
Memory problems we're seeing in using axis. What version of xerces is  
axis distributed with (if any?)

--Peter

On Jul 19, 2004, at 9:57 AM, Davanum Srinivas wrote:

> Nishant,
>
> Switch ByteArray to use Commons' BOAS. Can you please try latest CVS
> and help us tune the impl? Are u willing to donate your BOAS to Apache
> for inclusion with Axis?
>
> thanks,
> dims
>
> On Sat, 17 Jul 2004 08:39:05 -0400, Davanum Srinivas  
> <davanum@gmail.com> wrote:
>> Nishant, Peter,
>>
>> Am willing to accept any patches against ByteArray incorporating the
>> concepts in your BOAS or commons BOAS. Since you have dived deep into
>> the code, can you please submit a patch? (please keep the file backup
>> code in case of REALLY big messages and please keep the existing flags
>> if possible.)
>>
>> As u said earlier, it is not the cause of the OOM of peter. and am
>> still unable to recreate it. I used JMeter for client-side and ran the
>> server on OptimizeIT. To set the stage i invoked the client once then
>> took a snapshot of the memory (after several forced gc's using the UI)
>> then i ran the client again and took a second snapshot (again after
>> several forced gc's using the UI) and i did not see any leaks.... :(
>>
>> thanks,
>> dims
>>
>>
>>
>> On 17 Jul 2004 15:10:56 +0530, Nishant Kumar  
>> <nishant.kumar@itellix.com> wrote:
>>> hi Peter,
>>>         i did some more tests in this regard and have attached some  
>>> interesting
>>> gc log images. let me first remove some small misunderstandings in my
>>> last mail.
>>>         i used POST just to reduce the cost at client. and i was  
>>> running
>>> simultaneous 8 clients. and all i wanted to say was that the server  
>>> (1.2
>>> beta) didn't end up with OOM.
>>>         secondly i insisted on using two separate machines just  
>>> because i don't
>>> have the luxury of 2GB machines.:(
>>>         now coming to you third point about the message size. i  
>>> totally agree
>>> with your point that the message size should be large to see this  
>>> error
>>> as soon as possible and i also think you have done a great job  
>>> writing
>>> out such a good test case. but it somehow doesnot result in OOM.
>>>
>>> so finally what have i done.
>>>
>>>         firstly i figured out a big memory hogger. the ByteArray  
>>> class.
>>> axis team could have chosen to use
>>> http://jakarta.apache.org/commons/io/apidocs/org/apache/commons/io/ 
>>> output/ByteArrayOutputStream.html
>>> or my own (having same functionality)
>>> http://nishantkumar.com/mycode/codedocs/nkio/com/nk/io/ 
>>> ByteArrayOutputStream.html
>>> instead. javadocs explains the difference. so i replaced the byte  
>>> array
>>> class with the second class mentioned above.
>>>
>>>         to see the memory usage pattern i ran your client in a  
>>> single thread. i
>>> captured the verbose gc output and viewed the same using gcviewer.
>>> memory allocated was 128M.
>>>         the first observation is that the unaltered code causes many  
>>> full GC to
>>> run. you can see the same in gc2.png.
>>>         so i changed the class i mentioned above and ran the test  
>>> again and
>>> this time there were far less full gc. see gc8.png.
>>>         but as you can see in this graph, the graph keeps on  
>>> increasing till a
>>> full gc. i was expecting the graph to come down as soon one request  
>>> was
>>> handled and the next request began (i ran the client in one thread
>>> remember). the reason for this was that since the byte array within  
>>> the
>>> ByteArray (even the modified one) class has lasted for so many small  
>>> gc,
>>> it has been promoted to the old generation space and was waiting for  
>>> a
>>> full gc to reclaim it.
>>>         to counter this i ran the server with -Xconcgc, concurrent  
>>> gc. see the
>>> result in gc6.png. in this case the graph dipped after each request.
>>> note that i had run this test for a smaller time.
>>>         since i had to observe these things, i had to run this test  
>>> with single
>>> client. so what i can conclude from this limited test is that just
>>> because memory seems to be rising, it doesn't mean that there is  
>>> memory
>>> leak, only when a Full GC is unable to recover the same can we say  
>>> that
>>> there is some problem. System.gc() is not guaranteed to run gc, not  
>>> even
>>> a partial gc.
>>>         axis team should think about changing the ByteArray class,  
>>> if not for
>>> memory reason then because there are many bugs in the class. using  
>>> file
>>> based backup in this class is again not a good idea or not nicely
>>> implemented.
>>>         hope this helps.
>>> thanks,
>>> nishant
>>>
>>> On Fri, 2004-07-16 at 23:52, Peter Molettiere wrote:
>>>> It's fine to try this test using a simple unix POST command,  
>>>> however,
>>>> you must make your client send multiple simultaneous requests. We've
>>>> done a lot of testing to isolate this server side memory leak, and
>>>> found that it only occurs when multiple client requests are being
>>>> handled at the same time on the server. You should be able to run  
>>>> all
>>>> threads indefinitely without running into a server side OOM error.  
>>>> Just
>>>> being able to run all eight threads once doesn't prove the leak  
>>>> isn't
>>>> there.
>>>>
>>>> As far as your assertion that you must run the client on a different
>>>> machine than the server, I don't see why this is a requirement. I  
>>>> can
>>>> run the client with 1G of heap, and the server with 1G of heap, and
>>>> still have plenty of physical ram left over for the OS and other  
>>>> apps.
>>>> (Yes, I have an insane amount or memory in my machine.) The client  
>>>> and
>>>> the server both run in completely separate VMs, so there should be  
>>>> no
>>>> issue running the tests this way. If you still think there is a  
>>>> valid
>>>> reason for needing to run the test on separate machines, I'm all  
>>>> ears.
>>>>
>>>> As far as your last point about the size of the messages we're  
>>>> dealing
>>>> with. This test case is meant to display a server side memory leak,  
>>>> not
>>>> that serialization/deserialization in axis uses extreme amounts of
>>>> temporary memory (which it does, but we can live with that). The bug
>>>> (memory leak on the server) can be replicated with much smaller  
>>>> return
>>>> response messages, but it takes longer to notice. We notice pretty
>>>> quickly with our application, since we do send dynamic messages of  
>>>> this
>>>> size occasionally.
>>>>
>>>> If you run in a profiler, like JProfiler, you'll see the server  
>>>> memory
>>>> use growing very plainly.
>>>>
>>>> --Peter
>>>>
>>>> On Jul 15, 2004, at 10:09 PM, Nishant Kumar wrote:
>>>>
>>>>> hi Peter,
>>>>>     i tried looking into this matter. let me first explain what i  
>>>>> modified
>>>>> in your test case.
>>>>>
>>>>> firstly i replaced your java client by a simple UNIX POST command.
>>>>>
>>>>> something like this.
>>>>>
>>>>> POST -H 'Content-Type: text/xml; charset=utf-8' -H 'Accept:
>>>>> application/soap+xml, application/dime, multipart/related, text/*'  
>>>>> -H
>>>>> 'User-Agent: Axis/1.2beta2' -H 'Host: localhost:8080' -H
>>>>> 'Cache-Control:
>>>>> no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: ""'
>>>>> http://10.0.2.124:8080/axis/services/MemoryTester <getRoot.xml >
>>>>> response
>>>>>
>>>>> where getRoot.xml has the packet you used to send by your java  
>>>>> client.
>>>>>
>>>>> the benefit of doing this was that one can avoid having  
>>>>> deserialization
>>>>> cost at the client.
>>>>>
>>>>> i hope you were anyway running the client on a different machine  
>>>>> than
>>>>> the server. this is a MUST.
>>>>>
>>>>> you can run with verbosegc to find out how much memory the server  
>>>>> is
>>>>> actually taking. no need to run gc.
>>>>>
>>>>> after doing this i was able to run 8 such concurrent client with  
>>>>> just
>>>>> 300M set as -mx.
>>>>>
>>>>> another thing i noted was that the response was 13MB in size. that  
>>>>> is a
>>>>> huge response, that too when you are generating the same and not
>>>>> serving
>>>>> some static file.
>>>>>
>>>>> thanks,
>>>>> nishant
>>>>>
>>>>>
>>>>> On Wed, 2004-07-14 at 23:53, Peter Molettiere wrote:
>>>>>> I reported 1423, with the test case -- If there's anything I can
 
>>>>>> do to
>>>>>> help you out at all, please let me know. I'm more than happy to 

>>>>>> help.
>>>>>>
>>>>>> --Peter
>>>>>>
>>>>>> On Jul 14, 2004, at 6:21 AM, Davanum Srinivas wrote:
>>>>>>
>>>>>>> Glen,
>>>>>>>
>>>>>>> I'd REALLY like your help with the following:
>>>>>>> http://nagoya.apache.org/jira/browse/AXIS-1120
>>>>>>> (DefaultTypeMappingImpl
>>>>>>> is not compliant with JAX-RPC 1.1)
>>>>>>> http://nagoya.apache.org/jira/browse/AXIS-1423 (OutOfMemoryError
 
>>>>>>> in
>>>>>>> axis with multi-threaded usage)
>>>>>>> http://nagoya.apache.org/jira/browse/AXIS-1391 (severe memory
 
>>>>>>> leakage
>>>>>>> in server side service (with tomcat))
>>>>>>>
>>>>>>> 1120 involves exposing the currently internal flag
>>>>>>> (jaxrpc11Compliance) via the various command line utils and ant
>>>>>>> tasks.
>>>>>>> the other two 1423 and 1391 are severe problems with performance
 
>>>>>>> with
>>>>>>> test cases that recreate the problem.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> dims
>>>>>>>
>>>>>>> On Wed, 14 Jul 2004 08:44:33 -0400, Glen Daniels
>>>>>>> <glen@thoughtcraft.com> wrote:
>>>>>>>> dims:
>>>>>>>>
>>>>>>>> Can you please let us know which bugs those were?  I'd like
to  
>>>>>>>> take
>>>>>>>> another
>>>>>>>> brief look at them during beta-2.
>>>>>>>>
>>>>>>>> --Glen
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Davanum Srinivas" <davanum@gmail.com>
>>>>>>>> To: <axis-dev@ws.apache.org>
>>>>>>>> Sent: Wednesday, July 14, 2004 8:32 AM
>>>>>>>> Subject: Re: When is Axis 1.2 expected to be "final"?
>>>>>>>>
>>>>>>>> WSE 2.0 is FINAL now!!!! (see
>>>>>>>> http://msdn.microsoft.com/webservices/building/wse/). Also
 
>>>>>>>> changed
>>>>>>>> the
>>>>>>>> priority for the blockers to major.
>>>>>>>>
>>>>>>>> -- dims
>>>>>>>>
>>>>>>>> On Wed, 14 Jul 2004 13:32:24 +0200, Banck, Arent-Jan
>>>>>>>> <ajbanck@informatica.com> wrote:
>>>>>>>>> As .NET 2.0 is only a beta, 1.1 is the most recent .NET
 
>>>>>>>>> version.
>>>>>>>>>
>>>>>>>>> There are currently 5 cases marked as blocker in Jira.
If  
>>>>>>>>> these are
>>>>>>>>> not
>>>>>>>> blockers for 1.2 they should probably be given another priority.
>>>>>>>>>
>>>>>>>>> - Arent-Jan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Davanum Srinivas [mailto:davanum@gmail.com]
>>>>>>>>> Sent: Wednesday, July 14, 2004 1:22 PM
>>>>>>>>> To: axis-dev@ws.apache.org
>>>>>>>>> Subject: Re: When is Axis 1.2 expected to be "final"?
>>>>>>>>>
>>>>>>>>> Thomas,
>>>>>>>>>
>>>>>>>>> IF someone submits a test case that displays this problem
with
>>>>>>>>> latest .NET
>>>>>>>> 2.0 FINAL and Axis LATEST CVS, then am willing to hold up
the  
>>>>>>>> beta.
>>>>>>>> Not
>>>>>>>> otherwise.
>>>>>>>>>
>>>>>>>>> -- dims
>>>>>>>>>
>>>>>>>>> On Wed, 14 Jul 2004 10:07:48 +0200, Thomas Börkel  
>>>>>>>>> <tbo@ap-ag.com>
>>>>>>>>> wrote:
>>>>>>>>>> HI!
>>>>>>>>>>
>>>>>>>>>> Sorry, but how can Axis 1.2 going to be released,
if there are
>>>>>>>>>> severe
>>>>>>>>>> interop problems with .NET?
>>>>>>>>>>
>>>>>>>>>> http://nagoya.apache.org/jira/browse/AXIS-1308
>>>>>>>>>>
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Tom Jordahl [mailto:tomj@macromedia.com]
>>>>>>>>>>> Sent: Freitag, 9. Juli 2004 17:03
>>>>>>>>>>> To: 'axis-dev@ws.apache.org'
>>>>>>>>>>> Subject: RE: When is Axis 1.2 expected to be
"final"?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Real soon now....
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Tom Jordahl
>>>>>>>>>>> Macromedia Server Development
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: ricky_frost@peoplesoft.com
>>>>>>>>>>> [mailto:ricky_frost@peoplesoft.com]
>>>>>>>>>>> Sent: Thursday, July 08, 2004 1:18 PM
>>>>>>>>>>> To: axis-dev@ws.apache.org
>>>>>>>>>>> Subject: When is Axis 1.2 expected to be "final"?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> A project I'm looking at (ws-wsrp4j) just incorporated
the  
>>>>>>>>>>> 1.2
>>>>>>>>>>> beta
>>>>>>>>>>> and so I would like to know. I asked on that
list but they
>>>>>>>>>>> referred
>>>>>>>>>>> me here...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Davanum Srinivas - http://webservices.apache.org/~dims/
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Davanum Srinivas - http://webservices.apache.org/~dims/
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Davanum Srinivas - http://webservices.apache.org/~dims/
>>>>>>
>>>>
>>>
>>>
>>>
>>
>> --
>> Davanum Srinivas - http://webservices.apache.org/~dims/
>>
>
>
> -- 
> Davanum Srinivas - http://webservices.apache.org/~dims/


Mime
View raw message