brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Walraven <stefan.walra...@cs.kuleuven.be>
Subject Re: Problems with running "My Web Cluster Blueprint" on OpenStack
Date Mon, 30 Jun 2014 09:29:43 GMT
Hi Alex,

I will check out the jclouds properties.

Wrt our openstack setup, it is not managed by me and it is to support 
our research, so speeding it up has not the highest priority. When 
manually starting new VMs, I do a quick soft reboot and then the keys 
are immediately available. But when I do this while using brooklyn, then 
the brooklyn manager reports a failure.

Thanks,
Stefan

On 30/06/14 10:07, Alex Heneveld wrote:
>
> Hi Stefan,
>
> I think there is a property in jclouds where you can extend the 
> 600000ms = 10m timeout waiting for ssh after the cloud reports the VM 
> is up.  I'm not sure what it is though -- if you find it can you let 
> us know?
>
> cc: brooklyn experts in jclouds @andreaturli - can we pass jclouds 
> property overrides by defining these in the usual brooklyn way?
>
> Stefan, the other thing I would look at is speeding up your 
> openstack.  There is no good excuse I can think of for it reporting 
> the VM up but leaving it unreachable via ssh for 10m+ ! (In a 
> commercial world you'd be paying for the VM at this point!) Is that an 
> option?
>
> Best
> Alex
>
> 2014-06-30 00:31:54,950 WARN  jclouds.compute [user thread 2]: << 
> sockets [172.16.5.82:22] didn't open after 600000 MILLISECONDS
> 2014-06-30 00:31:54,951 DEBUG jclouds.compute [user thread 2]: << 
> customized node(RegionOne/01500f84-8c97-457d-9c84-c4baa5828660)
> 2014-06-30 00:31:54,962 ERROR b.l.jclouds.JcloudsLocation 
> [brooklyn-execmanager-asXnC5iq-15]: Failed to start VM for 
> openstack-nova:< url >:5000/v2.0/@MySqlNodeImpl{id=Tknu8Hsu}: error 
> running 1 node group(brooklyn-zijh-< user 
> >-my-web-cluster-rddn-mysql-tknu) location(RegionOne) 
> image(9ff09af4-97e1-4c6c-aaa5-21f3ffdf8998) size(3) 
> options({inboundPorts=[22, 3306], scriptPresent=true, 
> userMetadata={Name=brooklyn-zijh-< user 
> >-my-web-cluster-rddn-mysql-tknu-VAsB}, autoAssignFloatingIp=false, 
> securityGroupNames=[brooklyn], keyPairName=defaultkey, 
> configDrive=false})
>
> END
>
> On 30/06/2014 01:27, Stefan Walraven wrote:
>> Hi Alex,
>>
>> Thanks! That destroyOnFailure property is indeed useful. I didn't see 
>> it in the spreadsheet with all the properties.
>>
>> You were right, the exception comes from jclouds, but it breaks the 
>> installation procedure by the Brooklyn manager. So the waitForSshable 
>> is not useful when the delay is long. I attached a part of the debug 
>> logs with the full stack trace (this stack trace is repeated for a 
>> few 1000 lines).
>>
>> The delay that the keys are active is quite long, it appears to be 
>> exact 1 hour... I think that the problem is within our OpenStack 
>> installation, because when I manually start VMs via the web console 
>> it also takes some time (but not that long...). But anyway, brooklyn 
>> fails after 10 minutes, every time again.
>>
>> Regards,
>> Stefan
>>
>> On 27/06/14 12:30, Alex Heneveld wrote:
>>
>>     Hi Stefan,
>>
>>     You can use this to keep failed VM's for post-mortems:
>>
>> brooklyn.location.named.MyOpenStack.destroyOnFailure=true
>>
>>     There is a debug log file which will report the IP address(es) it is
>>     trying to use.  Once the VM is up, can you also try SSH'ing and 
>> again
>>     confirm that this eventually works:
>>
>>          ssh -i /path/to/your-keypair-name.pem ubuntu@IP
>>
>>     If there's a delay let us know how long that is.
>>
>>     A few other things:
>>
>>     * If you don't specify keypair, any privateKey, or
>>     openstack-nova.auto-generate-keypairs=false, it should normally work
>>     (assuming openstack is configured so it installs and reports keys
>>     correctly), and Brooklyn will default to the jclouds behaviour of
>>     creating a user, using your local username, and installing a new
>>     public
>>     key for use, using ~/.ssh/id_rsa.pub if that exists -- so you can
>>     then
>>     just   `ssh IP`
>>
>>     * waitForReachable timeout can use a human-readable syntax, e.g. 
>> "1h"
>>     (the default is 5m)
>>
>>     * Can you send the full stack trace?  I think this timeout is from
>>     jclouds testing ssh which is a slightly different pathway as when
>>     this
>>     timeout fails it gives a message "SSH failed for ..." .  But
>>     definitely
>>     confusing, we should try to fix.
>>
>>     Best
>>     Alex
>>
>>     On 27/06/14 11:59, Stefan Walraven wrote:
>>>     Hi,
>>>
>>>     I'm experimenting with brooklyn on a private OpenStack
>>>     environment. But the deployment fails when brooklyn tries to
>>>     connect with the created VMs. I saw there was another thread
>>>     about this in the beginning of this month, but the proposed
>>>     solution doesn't work in my case.
>>>
>>>     Similar to the previous thread, the three VMs are launched, but
>>>     then it constantly fails when the brooklyn manager tries to ssh
>>>     the VMs:
>>>
>>>     java.util.NoSuchElementException: could not connect to any ip
>>>     address port 22 on node
>>>
>>>
>>>     I have a keypair available in my OpenStack installation (added
>>>     via web console), and I specified in brooklyn.properties to use
>>>     that keypair. It effectively uses that one, but it still fails to
>>>     login. Yesterday, I tried to login myself to those VMs using SSH
>>>     and it didn't work immediately (there is some delay), but after a
>>>     while I got in, so the key pair is installed properly. However, I
>>>     can't test this any more, because now brooklyn immediately
>>>     removes the VMs after failure (which it didn't yesterday...).
>>>
>>>     Anyway, I tried to solve this issue using the waitForSshable
>>>     property, but apparently this has no effect. The failure always
>>>     occurs after the same period of time (around 10 minutes after the
>>>     process is started), even if I put the waitForSshable property on
>>>     1 hour... My brooklyn.properties looks as follows:
>>>
>>>     brooklyn.location.named.MyOpenStack=< url >
>>>
>>>     brooklyn.location.named.MyOpenStack.identity=<tenant name / user >
>>>
>>>     brooklyn.location.named.MyOpenStack.credential=< password >
>>>
>>>     brooklyn.location.named.MyOpenStack.imageId=<region / id>
>>>
>>>     brooklyn.location.named.MyOpenStack.user=ubuntu
>>>
>>>     brooklyn.location.named.MyOpenStack.keyPair=defaultkey
>>>
>>> brooklyn.location.named.MyOpenStack.securityGroups=brooklyn
>>>
>>> brooklyn.location.named.MyOpenStack.jclouds.openstack-nova.auto-create-floating-ips=false
>>>
>>>
>>> brooklyn.location.named.MyOpenStack.jclouds.openstack-nova.auto-generate-keypairs=false
>>>
>>>
>>>     brooklyn.location.named.MyOpenStack.waitForSshable=3600000 #
>>>     absurd high number but no effect
>>>
>>>     brooklyn.location.named.MyOpenStack.minRam=1024
>>>
>>>
>>>     I tried the solution suggested in the previous thread, But this
>>>     also results in the same issue:
>>>
>>> brooklyn.location.jclouds.my-openstack.keyPair=your-keypair-name
>>>
>>> brooklyn.location.jclouds.my-openstack.loginUser.privateKeyFile=/path/to/your-keypair-name.pem
>>>
>>>
>>>
>>>     Furthermore, I tried to upload my keys via brooklyn, but then I
>>>     don't get a key pair (similar to the previous thread). That only
>>>     works when using the keyPair property, but this requires the key
>>>     pair to be already available in the OpenStack installation (as
>>>     far as I experienced).
>>>
>>>     I hope someone can help me to solve this issue.
>>>
>>>     Thanks in advance,
>>>     Stefan Walraven
>>
>>
>>
>> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm for more 
>> information.
>
>


Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

Mime
View raw message