Return-Path: Delivered-To: apmail-ws-axis-user-archive@www.apache.org Received: (qmail 70112 invoked from network); 4 Jul 2007 06:06:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 4 Jul 2007 06:06:50 -0000 Received: (qmail 24946 invoked by uid 500); 4 Jul 2007 06:06:41 -0000 Delivered-To: apmail-ws-axis-user-archive@ws.apache.org Received: (qmail 24931 invoked by uid 500); 4 Jul 2007 06:06:40 -0000 Mailing-List: contact axis-user-help@ws.apache.org; run by ezmlm Precedence: bulk Reply-To: axis-user@ws.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list axis-user@ws.apache.org Received: (qmail 24920 invoked by uid 99); 4 Jul 2007 06:06:40 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jul 2007 23:06:40 -0700 X-ASF-Spam-Status: No, hits=2.5 required=10.0 tests=HTML_MESSAGE,NO_REAL_NAME,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of greg.lee-shoy@centrelink.gov.au designates 203.13.2.140 as permitted sender) Received: from [203.13.2.140] (HELO mailout2.centrelink.gov.au) (203.13.2.140) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jul 2007 23:06:35 -0700 In-Reply-To: <4688E9A1.8070808@wso2.com> To: axis-user@ws.apache.org Subject: Re: [Axis2] Temporary files/ Stress testing with Axis2 1.2 [SEC=UNCLASSIFIED] MIME-Version: 1.0 Message-ID: From: greg.lee-shoy@centrelink.gov.au Date: Wed, 4 Jul 2007 16:06:08 +1000 Content-Type: multipart/alternative; boundary="=_alternative 0021851CCA25730E_=" X-Virus-Checked: Checked by ClamAV on apache.org --=_alternative 0021851CCA25730E_= Content-Type: text/plain; charset="US-ASCII" Hello Asankha, At the moment we are running on Windows XP, so we'll have to do some research about how to increase the open file limit on Windows. We did some further testing on this issue, and found that the "Too many open files" error only seems to occur when hot deployment is set to true in axis2.xml, and it occurs after ~400 iterations of loading the configurationContext. Also, even with the patch we put into the org.apache.axis2.deployment.util.Utils class, another problem (OutOfMemoryError) occurs after ~2826 iterations - so it's not really a permanent fix. When hot deployment is set to false however, no problems seem to occur even after 100000 iterations. I suspect this might be related to the issue raised previously in this user forum regarding cleanup of AxisConfigurators (see http://mail-archives.apache.org/mod_mbox/ws-axis-user/200706.mbox/%3c19e0530f0706190840n52ac1b17p93664fa91e792a8b@mail.gmail.com%3e ). Also, when we substituted the nightly build from 3rd July into the dependencies for the test (leaving hotdeployment as true), we get the original "Too many open files" error after 400 iterations. I've created a issue in JIRA for this ( https://issues.apache.org/jira/browse/AXIS2-2899) and attached the configuration/modules we're using to the issue. Thanks, Greg "Asankha C. Perera" 02/07/2007 10:03 PM Please respond to axis-user@ws.apache.org To axis-user@ws.apache.org cc Subject Re: [Axis2] Temporary files/ Stress testing with Axis2 1.2 [SEC=UNCLASSIFIED] Reference Expires Greg Are you running the system on Windows or Linux? The OS decides the number of files a process can open. If you are using a Unix system, try to increase the number of files as follows. This is not a solution for a possible Axis2 issue, but something to make sure before you run a typical load test. You may be able to find alternatives for Windows on Google. Maybe http://coldfused.blogspot.com/2007/02/mystery-of-too-many-open-files.html would be of interest too. /etc/sysctl.conf fs.file-max = 2097152 /etc/security/limits.conf * soft nofile 4096 * hard nofile 65535 asankha Davanum Srinivas wrote: > Greg, > > Please create an issue in JIRA and upload your test ASAP. we are in > the middle of perf testing. this would be a great scenario to fix for > 1.3 release. > > thx, > dims > > On 7/2/07, greg.lee-shoy@centrelink.gov.au > wrote: >> >> Hello, >> >> Our team is currently using Axis2 1.2 in our application and we are >> running >> some performance and stress tests with JMeter. >> While running these stress tests we have come across two issues that >> we hope >> someone can help us with. >> >> The first issue relates to temporary files. In our stress testing we >> were >> using the URLBasedAxisConfigurator to load .mar files on the client >> side, >> but got the error below with 1 user and 1000 iterations: >> >> java.lang.RuntimeException: java.io.IOException: Too many open files;; >> at >> org.apache.axis2.deployment.util.Utils.getURLsForAllJars(Utils.java(Compiled >> >> Code));; >> at >> org.apache.axis2.deployment.util.Utils.createClassLoader(Utils.java:684);; >> >> at >> au.gov.centrelink.jfs.proxy.axis2.ClientURLBasedAxisConfigurator.loadRepositoryFromURL(ClientURLBasedAxisConfigurator.java(Compiled >> >> Code));; >> at >> org.apache.axis2.deployment.URLBasedAxisConfigurator.getAxisConfiguration(URLBasedAxisConfigurator.java:72);; >> >> at >> org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:60);; >> >> at >> au.gov.centrelink.jfs.proxy.ProxyFactory.createStub(ProxyFactory.java:699);; >> >> at >> au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:440);; >> >> at >> au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:395);; >> >> at >> au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java:64);; >> >> at >> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421);; >> >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226);; >> >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);; >> >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:763);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:856);; >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);; >> >> at >> com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:334);; >> >> at >> org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056);; >> >> at >> org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388);; >> >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231);; >> >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);; >> >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:763);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:856);; >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:136);; >> >> at >> au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java:60);; >> >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);; >> >> at >> au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java:267);; >> >> at >> au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java:99);; >> >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);; >> >> at >> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:89);; >> >> at >> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1924);; >> >> at >> com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:98);; >> >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:472);; >> >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:411);; >> >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:288);; >> >> at >> com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:207);; >> >> at >> com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:109);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:566);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:619);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:952);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1039);; >> >> at >> com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1471);; >> >> Caused by: java.io.IOException: Too many open files;; >> at >> java.io.WinNTFileSystem.createFileExclusively(Native >> Method);; >> at java.io.File.checkAndCreate(File.java(Compiled Code));; >> at java.io.File.createTempFile(File.java(Compiled Code));; >> at java.io.File.createTempFile(File.java(Inlined Compiled >> Code));; >> at >> org.apache.axis2.deployment.util.Utils.createTempFile(Utils.java(Compiled >> >> Code));; ... 53 more;; ]] >> >> >> At that time we were loading the axis2 Configuration context each >> time the >> Stub was instantiated. Looking at the code for the method >> getURLsForAllJars() in >> org.apache.axis2.deployment.util.Utils, it looks like a >> temp file is always created for the URL that is passed in. Is it >> possible >> to change the method so it checks whether the URL uses the 'file' >> protocol, >> and if so, read from the file directly, instead of creating a temp >> file and >> reading from it? When we made this change we found that the "Too >> many open >> files" problem seemed to go away. Although we have switched to >> caching the >> configuration context rather than loading it every time, we still >> think the >> above change would be a good one to make. >> >> >> Our other issue is a WstxUnexpectedCharException that seems to occur >> intermittently - it has occurred only once in our last two runs. >> We were running JMeter with 20 concurrent users, each of which makes >> 10,000 >> requests (200,000 requests made in total) to a web application. The web >> application acts as the web service client and makes service calls to a >> servlet. The servlet returns essentially the same SOAP response for >> every >> request it receives (the only change is in the text content of an xml >> element in the SOAP body of the response, which changes slightly each >> time). >> In the last two runs, we got one instance of the following >> exception, which >> seems to happen in the web app while parsing the SOAP response: >> >> [29/06/07 13:27:14:469 EST] 0000004a SystemErr R >> au.gov.centrelink.jfs.proxy.envlookup.EnvironmentLookupException: >> RemoteException occurred while doing an environment lookup for CRN >> 200393036S >> at >> au.gov.centrelink.jfs.proxy.envlookup.impl.DefaultCustomerEnvironmentLookupImpl.findEnvironment(DefaultCustomerEnvironmentLookupImpl.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient.doRouting(AbstractCustomerServiceClient.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.CGLIB$doRouting$0((Inlined >> >> Compiled Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46$$FastClassByCGLIB$$183db92b.invoke((Compiled >> >> Code)) >> at >> net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java(Inlined >> Compiled Code)) >> at >> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.doRouting((Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.retrieveEmailAddress((Compiled >> >> Code)) >> at >> au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(Inlined >> >> Compiled Code)) >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined >> >> Compiled Code)) >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled >> >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined >> >> Compiled Code)) >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled >> >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled >> >> Code)) >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled >> >> Code)) >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled >> >> Code)) >> at >> com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled >> Code)) >> Caused by: org.apache.axis2.AxisFault: >> com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected >> character 'E' (code 69) in prolog; expected '<' >> at [row,col {unknown-source}]: [1,1] >> at >> org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled >> >> Code)) >> at >> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java(Compiled >> >> Code)) >> at >> org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.isislocation.service.IsisLocationStub.envLookup(IsisLocationStub.java(Compiled >> >> Code)) >> at >> sun.reflect.GeneratedMethodAccessor40.invoke(Unknown >> Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled >> >> Code)) >> at java.lang.reflect.Method.invoke(Method.java(Compiled Code)) >> at >> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractDirectServiceClient$$EnhancerByCGLIB$$b007db1b.envLookup((Compiled >> >> Code)) >> ... 52 more >> Caused by: org.apache.axiom.om.OMException: >> com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected >> character 'E' (code 69) in prolog; expected '<' >> at [row,col {unknown-source}]: [1,1] >> at >> org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java(Compiled >> >> Code)) >> at >> org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java(Inlined >> >> Compiled Code)) >> at >> org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.(StAXSOAPModelBuilder.java(Inlined >> >> Compiled Code)) >> at >> org.apache.axis2.builder.BuilderUtil.getSOAPBuilder(BuilderUtil.java(Inlined >> >> Compiled Code)) >> at >> org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled >> >> Code)) >> ... 61 more >> Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: >> Unexpected character 'E' (code 69) in prolog; expected '<' >> at [row,col {unknown-source}]: [1,1] >> at >> com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:623) >> >> at >> com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java(Compiled >> >> Code)) >> at >> com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java(Compiled >> Code)) >> ... 66 more >> >> >> It is strange that this exception occurs only once given that the >> servlet >> returns essentially the same SOAP response every time. >> Has anyone encountered this error before in a stress testing >> scenario, or >> has any idea what might cause it to occur intermittently? >> >> Thanks for your ongoing assistance, >> >> Greg********************************************************************** >> >> IMPORTANT: This e-mail is intended for the use of the addressee and may >> contain information that is confidential, commercially valuable or >> subject >> to legal or parliamentary privilege. If you are not the intended >> recipient >> you are notified that any review, re-transmission, disclosure, use or >> dissemination of this communication is strictly prohibited by several >> Commonwealth Acts of Parliament. If you have received this >> communication in >> error please notify the sender immediately and delete all copies of this >> transmission together with any attachments. >> ********************************************************************** > > --------------------------------------------------------------------- To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org For additional commands, e-mail: axis-user-help@ws.apache.org ********************************************************************** IMPORTANT: This e-mail is intended for the use of the addressee and may contain information that is confidential, commercially valuable or subject to legal or parliamentary privilege. If you are not the intended recipient you are notified that any review, re-transmission, disclosure, use or dissemination of this communication is strictly prohibited by several Commonwealth Acts of Parliament. If you have received this communication in error please notify the sender immediately and delete all copies of this transmission together with any attachments. ********************************************************************** --=_alternative 0021851CCA25730E_= Content-Type: text/html; charset="US-ASCII"
Hello Asankha,

At the moment we are running on Windows XP, so we'll have to do some research about how to increase the open file limit on Windows.  

We did some further testing on this issue, and found that the "Too many open files" error only seems to occur when hot deployment is set to true in axis2.xml, and it occurs after ~400 iterations of loading the configurationContext.  Also, even with the patch we put into the org.apache.axis2.deployment.util.Utils class, another problem (OutOfMemoryError) occurs after ~2826 iterations - so it's not really
a permanent fix.  

When hot deployment is set to false however, no problems seem to occur even after 100000 iterations.  

I suspect this might be related to the issue raised previously in this user forum regarding cleanup of AxisConfigurators (see http://mail-archives.apache.org/mod_mbox/ws-axis-user/200706.mbox/%3c19e0530f0706190840n52ac1b17p93664fa91e792a8b@mail.gmail.com%3e).  

Also, when we substituted the nightly build from 3rd July into the dependencies for the test (leaving hotdeployment as true), we get the original "Too many open files" error after 400 iterations.  
I've created a issue in JIRA for this (https://issues.apache.org/jira/browse/AXIS2-2899) and attached the configuration/modules we're using to the issue.

Thanks,
Greg




"Asankha C. Perera" <asankha@wso2.com>

02/07/2007 10:03 PM
Please respond to
axis-user@ws.apache.org
To
axis-user@ws.apache.org
cc
Subject
Re: [Axis2] Temporary files/ Stress testing with Axis2 1.2 [SEC=UNCLASSIFIED]
Reference
 Expires





Greg

Are you running the system on Windows or Linux? The OS decides the
number of files a process can open. If you are using a Unix system, try
to increase the number of files as follows. This is not a solution for a
possible Axis2 issue, but something to make sure before you run a
typical load test. You may be able to find alternatives for Windows on
Google. Maybe
http://coldfused.blogspot.com/2007/02/mystery-of-too-many-open-files.html
would be of interest too.

/etc/sysctl.conf
fs.file-max = 2097152

/etc/security/limits.conf
* soft nofile 4096
* hard nofile 65535

asankha

Davanum Srinivas wrote:
> Greg,
>
> Please create an issue in JIRA and upload your test ASAP. we are in
> the middle of perf testing. this would be a great scenario to fix for
> 1.3 release.
>
> thx,
> dims
>
> On 7/2/07, greg.lee-shoy@centrelink.gov.au
> <greg.lee-shoy@centrelink.gov.au> wrote:
>>
>> Hello,
>>
>> Our team is currently using Axis2 1.2 in our application and we are
>> running
>> some performance and stress tests with JMeter.
>> While running these stress tests we have come across two issues that
>> we hope
>> someone can help us with.
>>
>> The first issue relates to temporary files.  In our stress testing we
>> were
>> using the URLBasedAxisConfigurator to load .mar files on the client
>> side,
>> but got the error below with 1 user and 1000 iterations:
>>
>> java.lang.RuntimeException: java.io.IOException: Too many open files;;
>>         at
>> org.apache.axis2.deployment.util.Utils.getURLsForAllJars(Utils.java(Compiled
>>
>> Code));;
>>         at
>> org.apache.axis2.deployment.util.Utils.createClassLoader(Utils.java:684);;
>>
>>         at
>> au.gov.centrelink.jfs.proxy.axis2.ClientURLBasedAxisConfigurator.loadRepositoryFromURL(ClientURLBasedAxisConfigurator.java(Compiled
>>
>> Code));;
>>         at
>> org.apache.axis2.deployment.URLBasedAxisConfigurator.getAxisConfiguration(URLBasedAxisConfigurator.java:72);;
>>
>>         at
>> org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:60);;
>>
>>         at
>> au.gov.centrelink.jfs.proxy.ProxyFactory.createStub(ProxyFactory.java:699);;
>>
>>         at
>> au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:440);;
>>
>>         at
>> au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:395);;
>>
>>         at
>> au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java:64);;
>>
>>         at
>> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421);;
>>
>>         at
>> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226);;
>>
>>         at
>> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);;
>>
>>         at
>> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);;
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:763);;
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:856);;
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);;
>>
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);;
>>
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);;
>>
>>         at
>> com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:334);;
>>
>>         at
>> org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056);;
>>
>>         at
>> org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388);;
>>
>>         at
>> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231);;
>>
>>         at
>> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);;
>>
>>         at
>> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);;
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:763);;
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:856);;
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);;
>>
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:136);;
>>
>>         at
>> au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java:60);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);;
>>
>>         at
>> au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java:267);;
>>
>>         at
>> au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java:99);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);;
>>
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);;
>>
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);;
>>
>>         at
>> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:89);;
>>
>>         at
>> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1924);;
>>
>>         at
>> com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:98);;
>>
>>         at
>> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:472);;
>>
>>         at
>> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:411);;
>>
>>         at
>> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:288);;
>>
>>         at
>> com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:207);;
>>
>>         at
>> com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:109);;
>>
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:566);;
>>
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:619);;
>>
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:952);;
>>
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1039);;
>>
>>         at
>> com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1471);;
>>
>> Caused by: java.io.IOException: Too many open files;;
>>         at
>> java.io.WinNTFileSystem.createFileExclusively(Native
>> Method);;
>>         at java.io.File.checkAndCreate(File.java(Compiled Code));;
>>         at java.io.File.createTempFile(File.java(Compiled Code));;
>>         at java.io.File.createTempFile(File.java(Inlined Compiled
>> Code));;
>>         at
>> org.apache.axis2.deployment.util.Utils.createTempFile(Utils.java(Compiled
>>
>> Code));; ... 53 more;; ]]
>>
>>
>> At that time we were loading the axis2 Configuration context each
>> time the
>> Stub was instantiated.  Looking at the code for the method
>> getURLsForAllJars() in
>> org.apache.axis2.deployment.util.Utils, it looks like a
>> temp file is always created for the URL that is passed in.  Is it
>> possible
>> to change the method so it checks whether the URL uses the 'file'
>> protocol,
>> and if so, read from the file directly, instead of creating a temp
>> file and
>> reading from it?  When we made this change we found that the "Too
>> many open
>> files" problem seemed to go away.  Although we have switched to
>> caching the
>> configuration context rather than loading it every time, we still
>> think the
>> above change would be a good one to make.
>>
>>
>> Our other issue is a WstxUnexpectedCharException that seems to occur
>> intermittently - it has occurred only once in our last two runs.
>> We were running JMeter with 20 concurrent users, each of which makes
>> 10,000
>> requests (200,000 requests made in total) to a web application.  The web
>> application acts as the web service client and makes service calls to a
>> servlet.  The servlet returns essentially the same SOAP response for
>> every
>> request it receives (the only change is in the text content of an xml
>> element in the SOAP body of the response, which changes slightly each
>> time).
>>  In the last two runs, we got one instance of the following
>> exception, which
>> seems to happen in the web app while parsing the SOAP response:
>>
>> [29/06/07 13:27:14:469 EST] 0000004a SystemErr     R
>> au.gov.centrelink.jfs.proxy.envlookup.EnvironmentLookupException:
>> RemoteException occurred while doing an environment lookup for CRN
>> 200393036S
>>         at
>> au.gov.centrelink.jfs.proxy.envlookup.impl.DefaultCustomerEnvironmentLookupImpl.findEnvironment(DefaultCustomerEnvironmentLookupImpl.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient.doRouting(AbstractCustomerServiceClient.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.CGLIB$doRouting$0(<generated>(Inlined
>>
>> Compiled Code))
>>         at
>> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46$$FastClassByCGLIB$$183db92b.invoke(<generated>(Compiled
>>
>> Code))
>>         at
>> net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java(Inlined
>> Compiled Code))
>>         at
>> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.doRouting(<generated>(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.retrieveEmailAddress(<generated>(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java(Compiled
>>
>> Code))
>>         at
>> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(Inlined
>>
>> Compiled Code))
>>         at
>> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled
>>
>> Code))
>>         at
>> org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined
>>
>> Compiled Code))
>>         at
>> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled
>>
>> Code))
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
>> Code))
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java(Compiled
>>
>> Code))
>>         at
>> org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java(Compiled
>>
>> Code))
>>         at
>> org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java(Compiled
>>
>> Code))
>>         at
>> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled
>>
>> Code))
>>         at
>> org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined
>>
>> Compiled Code))
>>         at
>> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled
>>
>> Code))
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
>> Code))
>>         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled
>>
>> Code))
>>         at
>> com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled
>> Code))
>> Caused by: org.apache.axis2.AxisFault:
>> com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected
>> character 'E' (code 69) in prolog; expected '<'
>>  at [row,col {unknown-source}]: [1,1]
>>         at
>> org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled
>>
>> Code))
>>         at
>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java(Compiled
>>
>> Code))
>>         at
>> org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.isislocation.service.IsisLocationStub.envLookup(IsisLocationStub.java(Compiled
>>
>> Code))
>>         at
>> sun.reflect.GeneratedMethodAccessor40.invoke(Unknown
>> Source)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled
>>
>> Code))
>>         at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
>>         at
>> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled
>>
>> Code))
>>         at
>> au.gov.centrelink.jfs.proxy.impl.AbstractDirectServiceClient$$EnhancerByCGLIB$$b007db1b.envLookup(<generated>(Compiled
>>
>> Code))
>>         ... 52 more
>> Caused by: org.apache.axiom.om.OMException:
>> com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected
>> character 'E' (code 69) in prolog; expected '<'
>>  at [row,col {unknown-source}]: [1,1]
>>         at
>> org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java(Compiled
>>
>> Code))
>>         at
>> org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java(Inlined
>>
>> Compiled Code))
>>         at
>> org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.<init>(StAXSOAPModelBuilder.java(Inlined
>>
>> Compiled Code))
>>         at
>> org.apache.axis2.builder.BuilderUtil.getSOAPBuilder(BuilderUtil.java(Inlined
>>
>> Compiled Code))
>>         at
>> org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled
>>
>> Code))
>>         ... 61 more
>> Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException:
>> Unexpected character 'E' (code 69) in prolog; expected '<'
>>  at [row,col {unknown-source}]: [1,1]
>>         at
>> com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:623)
>>
>>         at
>> com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java(Compiled
>>
>> Code))
>>         at
>> com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java(Compiled
>> Code))
>>         ... 66 more
>>
>>
>> It is strange that this exception occurs only once given that the
>> servlet
>> returns essentially the same SOAP response every time.
>> Has anyone encountered this error before in a stress testing
>> scenario, or
>> has any idea what might cause it to occur intermittently?
>>
>> Thanks for your ongoing assistance,
>>
>> Greg**********************************************************************
>>
>> IMPORTANT: This e-mail is intended for the use of the addressee and may
>> contain information that is confidential, commercially valuable or
>> subject
>> to legal or parliamentary privilege. If you are not the intended
>> recipient
>> you are notified that any review, re-transmission, disclosure, use or
>> dissemination of this communication is strictly prohibited by several
>> Commonwealth Acts of Parliament. If you have received this
>> communication in
>> error please notify the sender immediately and delete all copies of this
>> transmission together with any attachments.
>> **********************************************************************
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org
********************************************************************** IMPORTANT: This e-mail is intended for the use of the addressee and may contain information that is confidential, commercially valuable or subject to legal or parliamentary privilege. If you are not the intended recipient you are notified that any review, re-transmission, disclosure, use or dissemination of this communication is strictly prohibited by several Commonwealth Acts of Parliament. If you have received this communication in error please notify the sender immediately and delete all copies of this transmission together with any attachments. ********************************************************************** --=_alternative 0021851CCA25730E_=--