brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (BROOKLYN-533) AWS VM deletion failed with "Request limit exceeded"
Date Mon, 18 Sep 2017 10:26:00 GMT

     [ https://issues.apache.org/jira/browse/BROOKLYN-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aled Sage updated BROOKLYN-533:
-------------------------------
    Description: 
I deployed an app with approx 100 VMs in AWS.

I then stopped my app, thus terminating all the VMs. However, some requests failed with response
{{503}}, {{RequestLimitExceeded}}. Those VMs were left running.

[EDIT] jclouds did do an exponential backoff, but because of the heavy load it still sometimes
failed for the 6 retries.

The propagated exception is shown below:
{noformat}
2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Problem releasing
machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-0663997ccc85af459;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
        at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
{noformat}

---
Searching the log, I also saw the error response shown below:

{noformat}
2017-09-12T20:02:08,902 ERROR 127 o.a.b.l.j.JcloudsLocation [er-yR7aVpU0-1224] Problem releasing
machine SshMachineLocation[52.211.119.194:amp@ec2-52-211-119-194.eu-west-1.compute.amazonaws.com/52.211.119.194:22(id=usug6ourid)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-074d162dceaf06b4f;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='7a0838cd-cbdc-49e1-95aa-6d3794b15839',
requestToken='null', code='Unavailable', message='The service is unavailable. Please try again
shortly.', context='{Response=, Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
failed with code 503, error: AWSError{requestId='7a0838cd-cbdc-49e1-95aa-6d3794b15839', requestToken='null',
code='Unavailable', message='The service is unavailable. Please try again shortly.', context='{Response=,
Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
        at com.sun.proxy.$Proxy201.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
{noformat}

[EDIT] Should jclouds have retried when it go the {{Unavailable}} code? Or was it right to
just propagate the exception?


  was:
I deployed an app with approx 100 VMs in AWS.

I then stopped my app, thus terminating all the VMs. However, some requests failed with response
{{503}}, {{RequestLimitExceeded}}. Those VMs were left running.

My opinion is that jclouds should have done an exponential backoff, to retry the instance
deletion.

The propagated exception is shown below:
{noformat}
2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Problem releasing
machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-0663997ccc85af459;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
        at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
{noformat}

---
Searching the log, I also saw the error response shown below:

{noformat}
2017-09-12T20:02:08,902 ERROR 127 o.a.b.l.j.JcloudsLocation [er-yR7aVpU0-1224] Problem releasing
machine SshMachineLocation[52.211.119.194:amp@ec2-52-211-119-194.eu-west-1.compute.amazonaws.com/52.211.119.194:22(id=usug6ourid)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-074d162dceaf06b4f;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='7a0838cd-cbdc-49e1-95aa-6d3794b15839',
requestToken='null', code='Unavailable', message='The service is unavailable. Please try again
shortly.', context='{Response=, Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
failed with code 503, error: AWSError{requestId='7a0838cd-cbdc-49e1-95aa-6d3794b15839', requestToken='null',
code='Unavailable', message='The service is unavailable. Please try again shortly.', context='{Response=,
Errors=}'}
        at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
        at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
        at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
        at com.sun.proxy.$Proxy201.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
        at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
        at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
        at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
        at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
        at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
{noformat}

What is the best thing for jclouds to do if it gets a response 503 {{RequestLimitExceeded}}
or a {{code='Unavailable', message='The service is unavailable. Please try again shortly.'}}?
Should it try again shortly (i.e. exponential backoff)? Or just propagate the exception? The
first feels like a definite retry; the second probably retry as well, though it's unclear
how long the service will be unavailable.



> AWS VM deletion failed with "Request limit exceeded"
> ----------------------------------------------------
>
>                 Key: BROOKLYN-533
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-533
>             Project: Brooklyn
>          Issue Type: Bug
>    Affects Versions: 0.11.0
>            Reporter: Aled Sage
>
> I deployed an app with approx 100 VMs in AWS.
> I then stopped my app, thus terminating all the VMs. However, some requests failed with
response {{503}}, {{RequestLimitExceeded}}. Those VMs were left running.
> [EDIT] jclouds did do an exponential backoff, but because of the heavy load it still
sometimes failed for the 6 retries.
> The propagated exception is shown below:
> {noformat}
> 2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Problem
releasing machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-0663997ccc85af459;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
> org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=,
Errors=}'}
>         at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
>         at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
>         at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
>         at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
>         at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
>         at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
>         at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
>         at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
>         at java.lang.Thread.run(Thread.java:748) [?:?]
> {noformat}
> ---
> Searching the log, I also saw the error response shown below:
> {noformat}
> 2017-09-12T20:02:08,902 ERROR 127 o.a.b.l.j.JcloudsLocation [er-yR7aVpU0-1224] Problem
releasing machine SshMachineLocation[52.211.119.194:amp@ec2-52-211-119-194.eu-west-1.compute.amazonaws.com/52.211.119.194:22(id=usug6ourid)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-074d162dceaf06b4f;
ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='7a0838cd-cbdc-49e1-95aa-6d3794b15839',
requestToken='null', code='Unavailable', message='The service is unavailable. Please try again
shortly.', context='{Response=, Errors=}'}
> org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error: AWSError{requestId='7a0838cd-cbdc-49e1-95aa-6d3794b15839',
requestToken='null', code='Unavailable', message='The service is unavailable. Please try again
shortly.', context='{Response=, Errors=}'}
>         at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
>         at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
>         at com.sun.proxy.$Proxy201.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
>         at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
>         at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
>         at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
>         at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
>         at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
>         at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
>         at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
>         at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
>         at java.lang.Thread.run(Thread.java:748) [?:?]
> {noformat}
> [EDIT] Should jclouds have retried when it go the {{Unavailable}} code? Or was it right
to just propagate the exception?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message