brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (BROOKLYN-424) Web-server hangs after several days
Date Thu, 27 Apr 2017 15:56:04 GMT

     [ https://issues.apache.org/jira/browse/BROOKLYN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aled Sage resolved BROOKLYN-424.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 0.11.0

> Web-server hangs after several days
> -----------------------------------
>
>                 Key: BROOKLYN-424
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-424
>             Project: Brooklyn
>          Issue Type: Bug
>            Reporter: Aled Sage
>             Fix For: 0.11.0
>
>
> Using Brooklyn 0.10.0-SNAPSHOT from (approximately) 13-09-16...
> The web-server becomes unresponsive. Refreshing the page doesn't help. 
> There are (probably) approx 100 catalog items in the persisted state, but no apps deployed.
> Looking at the jstack output, there are 16 threads (all with names matching "brooklyn-jetty-server-8081-qtp97076093-.*")
that are parked on the same groovy LockableObject (0x00000007030f2d50).
> Most of the stacktraces looks something like:
> {noformat}
> "brooklyn-jetty-server-8081-qtp97076093-17830" prio=10 tid=0x00007f430c5db800 nid=0x1d4d
waiting on condition [0x00007f42c5cd8000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007030f2d50> (a org.codehaus.groovy.util.LockableObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
>         at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:34)
>         at org.codehaus.groovy.reflection.ClassInfo.lock(ClassInfo.java:327)
>         at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:239)
>         at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255)
>         at groovy.lang.MetaClassImpl.pickStaticMethod(MetaClassImpl.java:1511)
>         at groovy.lang.MetaClassImpl.retrieveStaticMethod(MetaClassImpl.java:1401)
>         at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1435)
>         at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:875)
>         at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:166)
>         at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.isCase(ScriptBytecodeAdapter.java:657)
>         at org.apache.brooklyn.util.groovy.GroovyJavaMethods.fix(GroovyJavaMethods.java:176)
>         at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
>         at org.apache.brooklyn.util.core.flags.FlagUtils.getFlagConfigKeyRecord(FlagUtils.java:277)
>         at org.apache.brooklyn.util.core.flags.FlagUtils.findAllFlagsAndConfigKeys(FlagUtils.java:250)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.findAllFlagsAndConfigKeyValues(BrooklynComponentTemplateResolver.java:378)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.configureEntityConfig(BrooklynComponentTemplateResolver.java:275)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.decorateSpec(BrooklynComponentTemplateResolver.java:255)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.populateSpec(BrooklynComponentTemplateResolver.java:246)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.resolveSpec(BrooklynComponentTemplateResolver.java:185)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.buildTemplateServicesAsSpecs(BrooklynAssemblyTemplateInstantiator.java:114)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.createServiceSpecs(BrooklynAssemblyTemplateInstantiator.java:73)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.createApplicationSpec(BrooklynAssemblyTemplateInstantiator.java:90)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.CampResolver.createEntitySpecFromServicesBlock(CampResolver.java:145)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.CampResolver.createSpecFromFull(CampResolver.java:109)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.CampToSpecTransformer.createCatalogSpec(CampToSpecTransformer.java:102)
>         at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog$1.apply(BasicBrooklynCatalog.java:331)
>         at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog$1.apply(BasicBrooklynCatalog.java:328)
>         at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:126)
>         at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:118)
>         at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.internalCreateSpecLegacy(BasicBrooklynCatalog.java:328)
>         at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.createSpec(BasicBrooklynCatalog.java:310)
>         at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogEntitySummary(CatalogTransformer.java:81)
>         at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogItemSummary(CatalogTransformer.java:117)
>         at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:91)
>         at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:88)
>         at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:582)
>         at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
>         at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContentsUsing(CollectionSerializer.java:139)
>         at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:87)
>         at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:23)
>         at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)
>         at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:250)
>         at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:604)
>         at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
>         at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1378)
>         at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:235)
>         at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:118)
>         at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:81)
>         at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
>         - locked <0x00000007dec2d290> (a org.apache.cxf.phase.PhaseInterceptorChain)
>         at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
>         at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
>         - locked <0x00000007dec17dc0> (a org.apache.cxf.phase.PhaseInterceptorChain)
>         at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
>         at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
>         at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
>         at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
>         at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
>         at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
>         at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
>         at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:217)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>         at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
>         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
>         at org.apache.brooklyn.rest.filter.LoggingFilter.doFilter(LoggingFilter.java:91)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>         at org.apache.brooklyn.rest.filter.RequestTaggingFilter.doFilter(RequestTaggingFilter.java:51)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>         at org.eclipse.jetty.server.Server.handle(Server.java:499)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And one looks like:
> {noformat}
> "brooklyn-jetty-server-8081-qtp97076093-16662" prio=10 tid=0x00007f430c0da000 nid=0x34ee
runnable [0x00007f431df67000]
>    java.lang.Thread.State: RUNNABLE
>         at java.util.Arrays.copyOf(Arrays.java:2367)
>         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>         at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>         at java.lang.StringBuilder.append(StringBuilder.java:132)
>         at java.lang.reflect.Method.toGenericString(Method.java:495)
>         at java.beans.MethodRef.find(MethodRef.java:79)
>         at java.beans.MethodRef.get(MethodRef.java:62)
>         at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:207)
>         - locked <0x0000000703139d30> (a java.beans.PropertyDescriptor)
>         at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2470)
>         at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
>         at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
>         at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
>         - locked <0x00000007defcff68> (a groovy.lang.MetaClassImpl)
>         at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
>         at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
>         at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255)
>         at groovy.lang.MetaClassImpl.pickStaticMethod(MetaClassImpl.java:1511)
>         at groovy.lang.MetaClassImpl.retrieveStaticMethod(MetaClassImpl.java:1401)
>         at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1435)
>         at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:875)
>         at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:166)
>         at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.isCase(ScriptBytecodeAdapter.java:657)
>         at org.apache.brooklyn.util.groovy.GroovyJavaMethods.fix(GroovyJavaMethods.java:176)
>         at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
>         at org.apache.brooklyn.core.effector.MethodEffector.<init>(MethodEffector.java:142)
>         at org.apache.brooklyn.core.effector.MethodEffector.<init>(MethodEffector.java:133)
>         at org.apache.brooklyn.core.effector.MethodEffector.create(MethodEffector.java:55)
>         at org.apache.brooklyn.core.entity.EntityDynamicType.findEffectors(EntityDynamicType.java:317)
>         at org.apache.brooklyn.core.entity.EntityDynamicType.<init>(EntityDynamicType.java:82)
>         at org.apache.brooklyn.core.entity.EntityDynamicType.<init>(EntityDynamicType.java:76)
>         at org.apache.brooklyn.core.objs.BrooklynTypes$ImmutableEntityType.<init>(BrooklynTypes.java:45)
>         at org.apache.brooklyn.core.objs.BrooklynTypes$DefinedBrooklynTypeLoader.call(BrooklynTypes.java:88)
>         at org.apache.brooklyn.core.objs.BrooklynTypes$DefinedBrooklynTypeLoader.call(BrooklynTypes.java:77)
>         at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
>         at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
>         at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
>         at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
>         - locked <0x00000007def8ed90> (a com.google.common.cache.LocalCache$WeakEntry)
>         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
>         at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
>         at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
>         at org.apache.brooklyn.core.objs.BrooklynTypes.getDefinedBrooklynType(BrooklynTypes.java:113)
>         at org.apache.brooklyn.core.objs.BasicSpecParameter$ParseClassParameters.parseParameters(BasicSpecParameter.java:358)
>         at org.apache.brooklyn.core.objs.BasicSpecParameter.fromSpec(BasicSpecParameter.java:168)
>         at org.apache.brooklyn.core.objs.BasicSpecParameter.addParameters(BasicSpecParameter.java:400)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynEntityDecorationResolver$SpecParameterResolver.decorate(BrooklynEntityDecorationResolver.java:213)
> {noformat}
> There is a discussion at http://groovy.329449.n5.nabble.com/Deadlock-on-concurrent-metaClass-access-tp3275702p3276031.html
(from 2010) about what could cause this.
> For us, we should *not* be calling groovy code here! I think that would get rid of the
problem.
> The results of {{top}} showed that the server was heavily loaded:
> {noformat}
> top - 11:57:51 up 25 days,  6:05,  2 users,  load average: 5.51, 5.02, 4.74
> Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
> Cpu(s): 13.3%us,  1.3%sy,  0.0%ni, 85.2%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
> Mem:   8061144k total,  5431376k used,  2629768k free,   195796k buffers
> Swap:  2097148k total,        0k used,  2097148k free,   626448k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
> 23196 brooklyn  20   0 7673m 3.9g  25m S 243.4 50.3  17960:32 java              
> 12577 brooklyn  20   0 15028 1124  816 R  3.6  0.0   0:00.40 top                
> ...
> {noformat}
> Looking at the 11 "RUNNABLE" threads in jstack (a pooor man's profiling!), only one was
in groovy code (shown below):
> {noformat}
> "brooklyn-jetty-server-8081-qtp97076093-16808" prio=10 tid=0x00007f430c0ff000 nid=0x7ca9
runnable [0x00007f4351cc2000]
>    java.lang.Thread.State: RUNNABLE
>         at java.lang.Class.getDeclaredMethods0(Native Method)
>         at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
>         at java.lang.Class.privateGetPublicMethods(Class.java:2743)
>         at java.lang.Class.getMethods(Class.java:1480)
>         at java.beans.MethodRef.find(MethodRef.java:77)
>         at java.beans.MethodRef.get(MethodRef.java:62)
>         at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:207)
>         - locked <0x000000070312ebb8> (a java.beans.PropertyDescriptor)
>         at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2470)
>         at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
>         at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
>         at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
>         - locked <0x00000007df006650> (a groovy.lang.MetaClassImpl)
>         at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
>         at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
>         at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:251)
>         at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:259)
>         at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:855)
>         at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:888)
>         at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:880)
>         at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToBoolean(DefaultTypeTransformation.java:180)
>         at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.booleanUnbox(DefaultTypeTransformation.java:69)
>         at org.apache.brooklyn.util.groovy.GroovyJavaMethods.truth(GroovyJavaMethods.java:149)
>         at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
>         at org.apache.brooklyn.util.core.flags.FlagUtils.getFlagConfigKeyRecord(FlagUtils.java:277)
>         at org.apache.brooklyn.util.core.flags.FlagUtils.findAllFlagsAndConfigKeys(FlagUtils.java:250)
>         at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.findAllFlagsAndConfigKeyValues(BrooklynComponentTemplateResolver.java:376)
>         ...
> {noformat}
> Memory usage was high (according to {{jmap -histo:live}}), but not unreasonable given
our use of soft references:
> {noformat}
>  num     #instances         #bytes  class name
> ----------------------------------------------
>    1:      14196525     1728415160  [C
> ...
> Total      73158949     3812605624
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message