brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BROOKLYN-534) ServiceRestarter policy provisioned new VM while entity was stopping
Date Mon, 18 Sep 2017 08:40:00 GMT
Aled Sage created BROOKLYN-534:
----------------------------------

             Summary: ServiceRestarter policy provisioned new VM while entity was stopping
                 Key: BROOKLYN-534
                 URL: https://issues.apache.org/jira/browse/BROOKLYN-534
             Project: Brooklyn
          Issue Type: Bug
    Affects Versions: 0.11.0
            Reporter: Aled Sage


I had an app consisting of approx 100 VMs. I called stop, to terminate all the VMs.

However, for one of the entities (nginx) it failed to stop the VM (see https://issues.apache.org/jira/browse/BROOKLYN-533).
Unfortunately the {{ServiceFailureDetector}} was still active, which caused the {{ServiceRestarter}}
to try to restart it! Because we'd partially shutdown nginx, it decided that a new machine
needed to be provisioned!

I presume that a fix would be to change {{ServiceRestarter}} so that it doesn't try to restart
it if the expected state is 'stopping' or 'stopped'. We should also check other policies (e.g.
{{ServiceReplacer}} to ensure they don't kick in if their associated entity is being stopped.

Snippets from the log are shown below:

{noformat}
2017-09-15T17:32:52,268 INFO  131 o.a.b.e.s.b.l.MachineLifecycleEffectorTasks [r-VlI23lev-80356]
Stopping NginxControllerImpl{id=t2ta5sv8ti} in [SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]]

2017-09-15T17:32:52,599 DEBUG 122 b.SSH [r-VlI23lev-80528] stopping NginxControllerImpl{id=t2ta5sv8ti},
initiating ssh on machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]:
#!/bin/bash -e

2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Problem releasing
machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-0663997ccc85af459;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
        at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
invalidating all entries in ssh pool cache. Final stats: CacheStats{hitCount=803, missCount=6,
loadSuccessCount=6, loadExceptionCount=0, totalLoadTime=1248584, evictionCount=4}
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
evicted from SshTool cache. Eviction cause: EXPLICIT
2017-09-15T17:34:09,966 DEBUG 143 o.a.b.u.p.BasicPool [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
closing, with 1 resources (currentLeased=0; totalLeased=789; totalCreated=1; totalClosed=0)
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] org.apache.brooklyn.location.ssh.SshMachineLocation$8@710df8f2
closing pool for aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com:22
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
evicted from SshTool cache. Eviction cause: EXPLICIT
2017-09-15T17:34:09,966 DEBUG 143 o.a.b.u.p.BasicPool [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
closing, with 1 resources (currentLeased=0; totalLeased=1; totalCreated=1; totalClosed=0)
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] org.apache.brooklyn.location.ssh.SshMachineLocation$8@6a0ceb61
closing pool for aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com:22
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager [r-VlI23lev-80548] org.apache.brooklyn.core.mgmt.internal.LocalLocationManager@68f8b22f
stopped management of location SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager [r-VlI23lev-80548] Missing
transition mode for SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
when unmanaging; assuming primary/destroying
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalUsageManager [r-VlI23lev-80548] Storing
location lifecycle usage event: location SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in state destroyed; caller context NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,967 INFO  127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Attempted release
of unknown machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager [r-VlI23lev-80548] Error
releasing SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in its parent JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]: java.lang.IllegalArgumentException:
Unknown machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.s.AttributeMap [r-VlI23lev-80548] removing attribute
internal.termination.task.state on NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,968 DEBUG 122 o.a.b.c.e.AbstractEntity [r-VlI23lev-80548] Emitting sensor
notification entity.sensor.removed value Sensor: internal.termination.task.state (org.apache.brooklyn.core.entity.internal.AttributesInternal$ProvisioningTaskState)
on NginxControllerImpl{id=t2ta5sv8ti}
...
2017-09-15T17:34:09,972 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} health-check for NginxControllerImpl{id=t2ta5sv8ti},
component now failing: location=[]; status=Present[value=on-fire]; lastPublished=NONE; timeNow=2017-09-15
17:34:09.972; currentFailurePeriod=<none> (stabilization 0ms); currentRecoveryPeriod=<none>
(stabilization 0ms)
2017-09-15T17:34:09,973 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} publishing failed
(state=Present[value=on-fire]; currentFailureStartTime=2017-09-15 17:34:09.972; now=2017-09-15
17:34:09.972
2017-09-15T17:34:09,973 DEBUG 122 o.a.b.c.e.AbstractEntity [r-VlI23lev-80386] Emitting sensor
notification ha.entityFailed value FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti},
description=service not up} on NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,973 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} setting on-fire, now
that deferred period has passed (state=Present[value=on-fire])
2017-09-15T17:34:09,973 INFO  129 o.a.b.p.h.ServiceRestarter [r-VlI23lev-80803] ServiceRestarter
notified; dispatching job for NginxControllerImpl{id=t2ta5sv8ti} (FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti},
description=service not up})
2017-09-15T17:34:09,974 WARN  129 o.a.b.p.h.ServiceRestarter [r-VlI23lev-80548] ServiceRestarter
acting on failure detected at NginxControllerImpl{id=t2ta5sv8ti} (FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti},
description=service not up})
2017-09-15T17:34:09,974 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80356] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} setting on-fire, now
that deferred period has passed (state=Present[value=on-fire])
2017-09-15T17:34:09,974 DEBUG 122 o.a.b.u.c.t.CompoundTask [r-VlI23lev-80356] Parent task
DstJob:pXCtsb5q ignoring child error (org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with
code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'})
in presence of our own error (org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with
code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'})
2017-09-15T17:34:09,974 DEBUG 122 o.a.b.c.e.Effectors [r-VlI23lev-80548] Replacing invocation
of restart[] on NginxControllerImpl{id=t2ta5sv8ti} with restart[restartChildren,restartMachine]
which is the impl defined at that entity
2017-09-15T17:34:09,974 WARN  122 o.a.b.c.m.i.EffectorUtils [r-VlI23lev-80356] Error invoking
stop at NginxControllerImpl{id=t2ta5sv8ti}: AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
2017-09-15T17:34:09,975 DEBUG 131 o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks
[r-VlI23lev-80386] restart of NginxControllerImpl{id=t2ta5sv8ti} has no hostname - doing machine-level
restart
2017-09-15T17:34:09,975 DEBUG 131 o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks
[r-VlI23lev-80386] restart of NginxControllerImpl{id=t2ta5sv8ti} requested be applied at machine
level
2017-09-15T17:34:09,975 DEBUG 131 o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks
[r-VlI23lev-80386] restart of NginxControllerImpl{id=t2ta5sv8ti} has no hostname - doing machine-level
restart
...
2017-09-15T17:34:09,975 DEBUG 122 o.a.b.c.m.i.EffectorUtils [r-VlI23lev-80356] Error invoking
stop at NginxControllerImpl{id=t2ta5sv8ti}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: 
        at org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129) [143:org.apache.brooklyn.utils-common:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStop(MachineLifecycleEffectorTasks.java:870)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStopLatching(MachineLifecycleEffectorTasks.java:757)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stop(MachineLifecycleEffectorTasks.java:744)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopEffectorBody.call(MachineLifecycleEffectorTasks.java:293)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopEffectorBody.call(MachineLifecycleEffectorTasks.java:290)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.core.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:82)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.util.concurrent.ExecutionException: org.jclouds.aws.AWSResponseException:
request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:?]
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
~[?:?]
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:382) ~[?:?]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStop(MachineLifecycleEffectorTasks.java:858)
~[?:?]
        ... 11 more
Caused by: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[?:?]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
~[?:?]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
~[?:?]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
~[?:?]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[?:?]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[?:?]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[?:?]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[?:?]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[?:?]
        at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) ~[?:?]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
~[?:?]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
~[?:?]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[?:?]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
~[?:?]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
~[?:?]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
~[?:?]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
~[?:?]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) ~[?:?]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
~[?:?]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
~[?:?]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
~[?:?]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
~[?:?]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
~[?:?]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
~[?:?]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
~[?:?]
        ... 6 more
2017-09-15T17:34:09,978 DEBUG 122 o.a.b.u.c.t.BasicExecutionManager [r-VlI23lev-80356] Exception
running task Task[stop]@pXCtsb5q (rethrowing): org.apache.brooklyn.core.mgmt.internal.EffectorUtils$EffectorCallPropagatedRuntimeException:
Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}: AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
2017-09-15T17:34:09,979 INFO  131 o.a.b.e.s.b.l.MachineLifecycleEffectorTasks [r-VlI23lev-80833]
Stopping NginxControllerImpl{id=t2ta5sv8ti} in []
...
2017-09-15T17:34:10,262 ERROR 107 o.j.a.h.AWSServerErrorRetryHandler [r-VlI23lev-81233] Cannot
retry after server error, command has exceeded retry limit 6: [method=org.jclouds.aws.ec2.features.AWSInstanceApi.public
abstract java.util.Set org.jclouds.aws.ec2.features.AWSInstanceApi.describeInstancesInRegion(java.lang.String,java.lang.String[])[eu-west-1,
[Ljava.lang.String;@3c00d5d0], request=POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1]
2017-09-15T17:34:10,263 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-81233] Problem releasing
machine SshMachineLocation[34.240.150.69:aled@ec2-34-240-150-69.eu-west-1.compute.amazonaws.com/34.240.150.69:22(id=dpxjea986f)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-02e2311ab1f16a0c8;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='e0e6ebbf-9a19-43ab-9219-25907eb8aa71',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
failed with code 503, error: AWSError{requestId='e0e6ebbf-9a19-43ab-9219-25907eb8aa71', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
        at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2DestroyNodeStrategy.destroyInstanceInRegion(AWSEC2DestroyNodeStrategy.java:59)
[47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2DestroyNodeStrategy.destroyNode(EC2DestroyNodeStrategy.java:72)
[77:ec2:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService$2.apply(BaseComputeService.java:298)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService$2.apply(BaseComputeService.java:295)
[100:jclouds-compute:2.0.2]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
...
2017-09-15T17:36:28,127 INFO  127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-81071] Creating VM
aws-ec2:eu-west-1@NginxControllerImpl{id=h0xas9pjk5} in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message