EC2 请求限制超出问题

Posted 2023-05-07

【中文标题】EC2 请求限制超出问题

【英文标题】：EC2 request limit exceeded issue

【发布时间】：2014-01-10 05:39:49

【问题描述】：

我正在运行每天启动（使用 JClouds）许多 AWS EC2 机器的自动化测试。 问题是有时机器由于“超出请求限制”错误而无法启动。

从下面的堆栈跟踪中，我认为我的操作失败了 5 次，而 5 是失败限制。也许这个动作是 jclouds 的方法 describeInstancesInRegion？为什么失败了？

我知道有一个 jclouds 错误，有时会以空/空名称启动机器。在 describeInstancesInRegion 调用之后会毁掉吗？

堆栈跟踪：

12/20/13 2:00:26 PM - Cannot retry after server error, **command has exceeded retry limit 5**: [method=public abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.aws.ec2.services.AWSInstanceAsyncClient.**describeInstancesInRegion**(java.lang.String,java.lang.String[])[eu-west-1, [Ljava.lang.String;@5987955b], request=POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1]
12/20/13 2:00:26 PM - << problem customizing node(eu-west-1/i-38cb4677): : Operation failed. org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSErrorrequestId='8937a5d2-f17f-4a42-af32-ed02125a73a4', requestToken='null', code='**RequestLimitExceeded**', message='Request limit exceeded.', context='Response=, Errors='
12/20/13 2:00:26 PM -   at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
12/20/13 2:00:26 PM -   at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
12/20/13 2:00:26 PM -   at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:180)
12/20/13 2:00:26 PM -   at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:150)
12/20/13 2:00:26 PM -   at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:131)
12/20/13 2:00:26 PM -   at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:97)
12/20/13 2:00:26 PM -   at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:58)
12/20/13 2:00:26 PM -   at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
12/20/13 2:00:26 PM -   at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
12/20/13 2:00:26 PM -   at $Proxy67.describeInstancesInRegion(Unknown Source)
12/20/13 2:00:26 PM -   at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:68)
12/20/13 2:00:26 PM -   at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:56)
12/20/13 2:00:26 PM -   at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:60)
12/20/13 2:00:26 PM -   at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:60)
12/20/13 2:00:26 PM -   at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
12/20/13 2:00:26 PM -   at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:59)
12/20/13 2:00:26 PM -   at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:42)
12/20/13 2:00:26 PM -   at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:148)
12/20/13 2:00:26 PM -   at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:143)
12/20/13 2:00:26 PM -   at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:160)
12/20/13 2:00:26 PM -   at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
12/20/13 2:00:26 PM -   at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:124)
12/20/13 2:00:26 PM -   at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:65)
12/20/13 2:00:26 PM -   at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:121)
12/20/13 2:00:26 PM -   at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:44)
12/20/13 2:00:26 PM -   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
12/20/13 2:00:26 PM -   at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
12/20/13 2:00:26 PM -   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
12/20/13 2:00:26 PM -   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
12/20/13 2:00:26 PM -   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
12/20/13 2:00:26 PM -   at java.lang.Thread.run(Thread.java:662)
12/20/13 2:00:26 PM - 
12/20/13 2:01:02 PM - Failed to create Cloudify Management VM: Error org.cloudifysource.esc.driver.provisioning.CloudProvisioningException: Failed to start machine.
12/20/13 2:00:26 PM - JClouds Deployer is shutting down a server that failed to start properly: eu-west-1/i-38cb4677
12/20/13 2:01:02 PM - Failed to start a management machine: Operation failed. java.util.concurrent.ExecutionException: org.cloudifysource.esc.driver.provisioning.CloudProvisioningException: Failed to start machine
12/20/13 2:01:02 PM -   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.driver.provisioning.BaseProvisioningDriver.doStartManagementMachines(BaseProvisioningDriver.java:327)
12/20/13 2:01:02 PM -   at java.util.concurrent.FutureTask.get(FutureTask.java:91)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.driver.provisioning.jclouds.DefaultProvisioningDriver.startManagementMachines(DefaultProvisioningDriver.java:427)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.shell.installer.CloudGridAgentBootstrapper.createManagementServers(CloudGridAgentBootstrapper.java:331)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.shell.installer.CloudGridAgentBootstrapper.getOrCreateManagementServers(CloudGridAgentBootstrapper.java:278)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.shell.commands.BootstrapCloud.doExecute(BootstrapCloud.java:224)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.shell.installer.CloudGridAgentBootstrapper.bootstrapCloudAndWait(CloudGridAgentBootstrapper.java:203)
12/20/13 2:01:02 PM -   at org.cloudifysource.shell.commands.AbstractGSCommand.execute(AbstractGSCommand.java:103)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)
12/20/13 2:01:02 PM -   at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)
12/20/13 2:01:02 PM -   at org.apache.karaf.shell.console.jline.Console.run(Console.java:172)
12/20/13 2:01:02 PM -   at org.apache.karaf.shell.console.Main.run(Main.java:191)
12/20/13 2:01:02 PM -   at org.apache.karaf.shell.console.Main.run(Main.java:89)
12/20/13 2:01:02 PM -   at org.cloudifysource.shell.GigaShellMain.main(GigaShellMain.java:126)
12/20/13 2:01:02 PM - Caused by: org.cloudifysource.esc.driver.provisioning.CloudProvisioningException: Failed to start machine
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.jclouds.JCloudsDeployer.createServer(JCloudsDeployer.java:178)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.driver.provisioning.jclouds.DefaultProvisioningDriver.createServer(DefaultProvisioningDriver.java:260)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.driver.provisioning.jclouds.DefaultProvisioningDriver.createServer(DefaultProvisioningDriver.java:236)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.driver.provisioning.BaseProvisioningDriver$1.call(BaseProvisioningDriver.java:315)
12/20/13 2:01:02 PM -   at org.cloudifysource.esc.driver.provisioning.BaseProvisioningDriver$1.call(BaseProvisioningDriver.java:310)
12/20/13 2:01:02 PM -   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
12/20/13 2:01:02 PM -   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
12/20/13 2:01:02 PM -   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
12/20/13 2:01:02 PM -   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
12/20/13 2:01:02 PM -   at java.lang.Thread.run(Thread.java:662)
12/20/13 2:01:02 PM - Caused by: org.jclouds.compute.RunNodesException: error running 1 node group(ec2-user-master-ec2gitservicestest-cloudify-manager1) location(eu-west-1) image(ami-c37474b7) size(m1.small) options(groupNames=[default], keyPair=ec2-sgtest-eu, userDataCksum=2f4a740b)

12/20/13 2:01:02 PM - Execution failures:
12/20/13 2:01:02 PM - 
12/20/13 2:01:02 PM - 0 error[s]
12/20/13 2:01:02 PM - Node failures:
12/20/13 2:01:02 PM - 
12/20/13 2:01:02 PM - 1) AWSResponseException on node eu-west-1/i-38cb4677:
12/20/13 2:01:02 PM - org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSErrorrequestId='8937a5d2-f17f-4a42-af32-ed02125a73a4', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='Response=, Errors='

jclouds version is: 1.6.2-incubating

【参考方案1】：

您需要告诉 jclouds 在轮询实例状态更新时不要经常发出请求。您可以通过以下覆盖来做到这一点。

Properties overrides = new Properties();
overrides.setProperty(POLL_INITIAL_PERIOD, POLL_PERIOD_TWENTY_SECONDS);
overrides.setProperty(POLL_MAX_PERIOD, POLL_PERIOD_TWENTY_SECONDS);

ComputeServiceContext context = ContextBuilder.newBuilder(PROVIDER)
        .credentials(username, apiKey)
        .overrides(overrides)
        .buildView(ComputeServiceContext.class);
computeService = context.getComputeService();

有关完整示例，请查看CloudServersPublish。虽然这是 Rackspace Cloud 的示例，但代码几乎相同。

【讨论】：

感谢重播。我认为它有所帮助，但只是部分帮助 - 这个错误仍然会发生，只是不太频繁。还有什么建议吗？

您是否尝试过延长投票周期？如果您要创建很多可能是问题的实例。我还建议启用logging。它将让您深入了解 jclouds 在网络上所做的事情。