whirr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Bayer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (WHIRR-455) Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
Date Mon, 25 Mar 2013 23:21:16 GMT

     [ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Bayer updated WHIRR-455:
-------------------------------

    Fix Version/s: 0.9.0
         Assignee: Andrew Bayer

Going to make a run at this for 0.9.0 - jclouds is working on improving request limiting,
but if that's not ready in time, I'll try to get some sort of staggering of instance creation.
                
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
>                 Key: WHIRR-455
>                 URL: https://issues.apache.org/jira/browse/WHIRR-455
>             Project: Whirr
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7.0
>         Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug  9 20:54:00
PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
>            Reporter: Bob Cotton
>            Assignee: Andrew Bayer
>             Fix For: 0.9.0
>
>         Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances,
and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't
see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue,
but I figured I'd start here.
> configuration file:
> {noformat}
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message