Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Thu, 5 Dec 2013 14:38:36 +0000 (UTC)
From: "Likitha Shetty (JIRA)" <jira@apache.org>
To: cloudstack-issues@incubator.apache.org
Message-ID: <JIRA.12682195.1386009859058.78113.1386254316772@arcas>
In-Reply-To: <JIRA.12682195.1386009859058@arcas>
References: <JIRA.12682195.1386009859058@arcas>
Subject: [jira] [Comment Edited] (CLOUDSTACK-5336) [Automation] During
 regression automation management server hang with "out of memory error"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CLOUDSTACK-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840100#comment-13840100 ] 

Likitha Shetty edited comment on CLOUDSTACK-5336 at 12/5/13 2:38 PM:
---------------------------------------------------------------------

Analysed the jmap dump to discover that too many VmwareContext objects are created and that is why there is a memory leak.
The above commit has debug statements which should help identify why exactly too many VmwareContexts are being created. Will reduce the log level to trace once this issue is fixed.


was (Author: likithas):
Analysed the jmap dump to discover that too many VmwareContext objects are created and that is why there is a memory leak.
This commit has debug statements which should help identify why exactly too many VmwareContexts are being created.   

> [Automation] During regression automation management server hang with "out of memory error" 
> --------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5336
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5336
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>         Environment: Vmware
> Branch : 4.3
>            Reporter: Rayees Namathponnan
>            Assignee: Likitha Shetty
>            Priority: Blocker
>             Fix For: 4.3.0
>
>         Attachments: CLOUDSTACK-5336.rar, dump.rar
>
>
> Steps to reproduce 
> Steps to reproduce 
> Create build from 4.3 branch and install from RPM
> Run BVT and Regression automation 
> Result 
> BVT complete without any issues; after start regression MS not responding, observed below "out of memory error"  in MS log 
> 2013-12-01 20:06:27,300 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-398:ctx-55572862 10.223.250.131) Unable to execute NetworkUsage command on DomR (10.223.250.17
> 4), domR may not be ready yet. failure due to Exception: java.lang.OutOfMemoryError
> Message: Java heap space
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:2219)
>         at java.util.ArrayList.grow(ArrayList.java:242)
>         at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216)
>         at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208)
>         at java.util.ArrayList.add(ArrayList.java:440)
>         at com.sun.xml.internal.ws.model.wsdl.WSDLOperationImpl.addFault(WSDLOperationImpl.java:126)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parsePortTypeOperationFault(RuntimeWSDLParser.java:740)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parsePortTypeOperation(RuntimeWSDLParser.java:727)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parsePortType(RuntimeWSDLParser.java:697)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parseWSDL(RuntimeWSDLParser.java:340)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parseImport(RuntimeWSDLParser.java:301)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parseImport(RuntimeWSDLParser.java:677)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parseWSDL(RuntimeWSDLParser.java:336)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:157)
>         at com.sun.xml.internal.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:120)
>         at com.sun.xml.internal.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:257)
>         at com.sun.xml.internal.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:220)
>         at com.sun.xml.internal.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:168)
>         at com.sun.xml.internal.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:96)
>         at javax.xml.ws.Service.<init>(Service.java:77)
>         at com.vmware.vim25.VimService.<init>(VimService.java:46)
>         at com.cloud.hypervisor.vmware.util.VmwareClient.connect(VmwareClient.java:129)
>         at com.cloud.hypervisor.vmware.resource.VmwareContextFactory.create(VmwareContextFactory.java:67)
>         at com.cloud.hypervisor.vmware.resource.VmwareContextFactory.getContext(VmwareContextFactory.java:85)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.getServiceContext(VmwareResource.java:6879)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.getServiceContext(VmwareResource.java:6861)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.networkUsage(VmwareResource.java:6578)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.getNetworkStats(VmwareResource.java:6596)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:672)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:505)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> 2013-12-01 20:06:27,409 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-32e68e7c) Found 1 routers to update status.
> 2013-12-01 20:06:27,467 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-32e68e7c) Found 0 networks to update RvR status.
> 2013-12-01 20:06:27,391 DEBUG [c.c.s.StorageManagerImpl] (StorageManager-Scavenger-4:ctx-e1826ff2) Storage pool garbage collector found 0 templates to clean up in storage pool: PS0
> Environment information
> [root@auto_ms2 management]# cat /etc/redhat-release
> CentOS release 6.2 (Final)
> [root@auto_ms2 management]# java -version
> java version "1.7.0_45"
> OpenJDK Runtime Environment (rhel-2.4.3.2.el6_4-x86_64 u45-b15)
> OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
> [root@auto_ms2 management]#
>  
> Attaching MS Log with this defect; i will create memory dump and attach with this defect  


--
This message was sent by Atlassian JIRA
(v6.1#6144)