cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nezar Madbouh <nezar.madb...@ipscape.com.au>
Subject Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Date Fri, 18 Mar 2016 01:19:05 GMT
Hi Simon,

I just noticed that from other Hosts I can't ssh into the SSVM:

[root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
root@169.254.3.161
ssh: connect to host 169.254.3.161 port 3922: No route to host


I can only ssh from one Host:
[root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
root@169.254.3.161
Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64

Thanks & Regards,
Nezar.

On 18 March 2016 at 10:36, Nezar Madbouh <nezar.madbouh@ipscape.com.au>
wrote:

> Hi Simon,
>
> Yes existing VMs are successfully running. Yesterday I was able to create
> a VM in Dev Environment but I can't create VMs in Prod environment.
>
> Thanks,
> Nezar.
>
> On 18 March 2016 at 09:12, Simon Weller <sweller@ena.com> wrote:
>
>> I'm assuming you have other VMs successfully running, correct? So you can
>> write to your primary storage.
>>
>>
>>
>> ________________________________________
>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
>> Sent: Thursday, March 17, 2016 4:43 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: Unable to create a New VM - Error message: Unable to start
>> instance due to Unable to get answer that is of class
>> com.cloud.agent.api.StartAnswer
>>
>> Hi Simon and Shweta,
>>
>> Please see the result below:
>>
>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>> ================================================
>> First DNS server is  8.8.8.8
>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>> --- 8.8.8.8 ping statistics ---
>> 2 packets transmitted, 2 packets received, 0% packet loss
>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>> Good: Can ping DNS server
>> ================================================
>> Good: DNS resolves download.cloud.com
>> ================================================
>> nfs is currently mounted
>> Mount point is /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>> Good: Can write to mount point
>> ================================================
>> Management server is 10.61.153.20. Checking connectivity.
>> Good: Can connect to management server port 8250
>> ================================================
>> Good: Java process is running
>> ================================================
>> Tests Complete. Look for ERROR or WARNING above.
>> root@s-11-VM:~#
>>
>> Thanks & Regards,
>> Nezar.
>>
>> On 17 March 2016 at 09:42, Simon Weller <sweller@ena.com> wrote:
>>
>> > Nezar,
>> >
>> > Log into the Cloudstack UI.
>> >
>> > Click on "Infrastructure" on the left menu. Click on  View All below
>> > "System VMs". find the item marked "Secondary Storage VM"  and then
>> click
>> > on the Name of that VM on the left (it will be s-XXXX-VM). You'll see a
>> > bunch of information about the VM. Find the Host line and also the Link
>> > Local IP Address line and then follow the original instructions in my
>> > previous email.
>> >
>> > - Si
>> >
>> >
>> >
>> >
>> >
>> > ________________________________________
>> > From: Shweta Agarwal <shweta.agarwal1@accelerite.com>
>> > Sent: Wednesday, March 16, 2016 12:16 AM
>> > To: users@cloudstack.apache.org
>> > Subject: Re: Unable to create a New VM - Error message: Unable to start
>> > instance due to Unable to get answer that is of class
>> > com.cloud.agent.api.StartAnswer
>> >
>> > Hi Nezar,
>> >
>> > You can find host name on detail view of SSVM in UI .
>> > You can also use list api command listSystemVms to get the host details
>> of
>> > system vms.
>> >
>> > Thanks
>> > Shweta
>> >
>> >
>> > ________________________________________
>> > From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
>> > Sent: Wednesday, March 16, 2016 9:58 AM
>> > To: users@cloudstack.apache.org
>> > Subject: Re: Unable to create a New VM - Error message: Unable to start
>> > instance due to Unable to get answer that is of class
>> > com.cloud.agent.api.StartAnswer
>> >
>> > Hi Simon,
>> >
>> > How can I find out which host the SSVM is on?
>> >
>> > Thanks & Regards,
>> > Nezar.
>> >
>> > On 16 March 2016 at 12:28, Simon Weller <sweller@ena.com> wrote:
>> >
>> > > Can you ssh into the Secondary Storage VM?
>> > >
>> > > This can be done by getting the link-local-ip address from the
>> > > infrastructure menu in the UI and then sshing into the host the SSVM
>> is
>> > on
>> > > and issuing this command as root: ssh -i /root/.ssh/id_rsa.cloud -p
>> 3922
>> > > root@link-local-ip  (replace link-local-ip with the link local ip).
>> > >
>> > > Try  running the SSVM diangnostic script:
>> > > usr/local/cloud/systemvm/ssvm-check.sh
>> > >
>> > > - Si
>> > > ________________________________________
>> > > From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
>> > > Sent: Tuesday, March 15, 2016 7:49 PM
>> > > To: users@cloudstack.apache.org
>> > > Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> > > instance due to Unable to get answer that is of class
>> > > com.cloud.agent.api.StartAnswer
>> > >
>> > > Hi Simon,
>> > >
>> > > Please see details below:
>> > >
>> > >
>> > > mysql> use cloud; select id, name, status, type from host where id =
>> 5;
>> > > Reading table information for completion of table and column names
>> > > You can turn off this feature to get a quicker startup with -A
>> > >
>> > > Database changed
>> > > +----+------------------------------+--------+---------+
>> > > | id | name                         | status | type    |
>> > > +----+------------------------------+--------+---------+
>> > > | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>> > > +----+------------------------------+--------+---------+
>> > > 1 row in set (0.00 sec)
>> > >
>> > >
>> > >
>> > > cat /var/log/cloudstack/agent/agent.log
>> > > com.cloud.utils.exception.CloudRuntimeException:
>> > > com.cloud.utils.exception.CloudRuntimeException: Can't find
>> > > volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>> > >         at
>> > >
>> > >
>> >
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>> > >         at
>> > >
>> > >
>> >
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>> > >         at
>> > >
>> > >
>> >
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>> > >         at
>> > >
>> > >
>> >
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>> > >         at com.cloud.agent.Agent.processRequest(Agent.java:501)
>> > >         at
>> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>> > >         at com.cloud.utils.nio.Task.run(Task.java:84)
>> > >         at
>> > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > >         at
>> > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > >         at java.lang.Thread.run(Thread.java:745)
>> > > agent.log.2016-03-03.gz (END)
>> > >
>> > >
>> > >
>> > > [root@eqx-cs-cmp-02 agent]# virsh pool-list
>> > > Name                 State      Autostart
>> > > -----------------------------------------
>> > > 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active     no
>> > > b3647480-5709-36eb-9ddd-e2c666f30b57 active     no
>> > > e3831246-ff55-4af4-9964-a17565edaccc active no
>> > >
>> > >
>> > >
>> > >
>> > > [Root @ eqx-cs-cmp-02 agent] # df -h
>> > > Filesystem            Size  Used Avail Use% Mounted on
>> > > /dev/mapper/vg_root-lv_root
>> > >                       203G 215G 1.3G 1% /
>> > > tmpfs 48g 48g 0 0% / dev / shm
>> > > / Dev / sda1 477m 427m 25M 6% / boot
>> > > xx.xx.xx.xx:/Cloudstack/primary
>> > >                       7.6T  4.1T  3.5T  54%
>> > > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>> > >
>> > >
>> > >
>> > >
>> > > I also noticed the following in cloudstack-agent.err
>> > >
>> > > libvirt: QEMU Driver error : Domain not found: no domain with matching
>> > name
>> > > 'i-11-600-VM'
>> > > libvirt: Storage Driver error : Storage volume not found: no storage
>> vol
>> > > with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>> > >
>> > > Your help is much appreciated,
>> > > Nezar
>> > >
>> > >
>> > > On 16 March 2016 at 11:03, Simon Weller <sweller@ena.com> wrote:
>> > >
>> > > > Nezar,
>> > > >
>> > > > ok, I just took a look at a pastebin you posted.
>> > > >
>> > > > I see it complaining about the inability to find a volume prior to
>> > > > throwing the host exception.
>> > > >
>> > > > Firstly, let's find out who host 5 is. Can you log into your MySQL
>> > server
>> > > > and run this:
>> > > >
>> > > > use cloud;select id,name,status,type from host where id=5;
>> > > >
>> > > > Can you post the results?
>> > > >
>> > > > If it's a host, can you post the /var/log/cloudstack/agent/agent.log
>> > from
>> > > > that host?
>> > > >
>> > > > And also run virsh pool-list on the host in question?
>> > > >
>> > > > - Si
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > ________________________________________
>> > > > From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
>> > > > Sent: Tuesday, March 15, 2016 6:11 PM
>> > > > To: users@cloudstack.apache.org
>> > > > Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> > > > instance due to Unable to get answer that is of class
>> > > > com.cloud.agent.api.StartAnswer
>> > > >
>> > > > Hi Simon,
>> > > >
>> > > > When you say "double check your routers tied to the network", do you
>> > mean
>> > > > the physical routers or the virtual routers?
>> > > >
>> > > > When I try to create a VM in Prod Environment it fails. I can't ping
>> > the
>> > > > Prod virtual router from management server or Host.
>> > > >
>> > > > But I can ping Dev virtual router from management server and Host
>> and I
>> > > was
>> > > > able to create a VM in Dev Environment.
>> > > >
>> > > > How can I troubleshoot the Prod Virtual router. I launched the
>> console
>> > > but
>> > > > couldn't login.
>> > > >
>> > > >
>> > > > Please note that we need to take into consideration that the
>> affected
>> > > > network is the Production network and we need to minimise service
>> > outage
>> > > as
>> > > > much as possible.
>> > > >
>> > > > Your help is much appreciated,
>> > > > Nezar
>> > > >
>> > > >
>> > > >
>> > > > On 16 March 2016 at 08:37, Simon Weller <sweller@ena.com> wrote:
>> > > >
>> > > > > I've seen this before where a Virtual Router has crashed and
the
>> > > > > management server can't communicate with it.
>> > > > > Are you using an advanced network setup?
>> > > > >
>> > > > > If so, double check your routers tied to the network where you
are
>> > > > > attempting to start an instance.
>> > > > >
>> > > > > - Si
>> > > > >
>> > > > >
>> > > > > ________________________________________
>> > > > > From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
>> > > > > Sent: Tuesday, March 15, 2016 4:02 PM
>> > > > > To: users@cloudstack.apache.org
>> > > > > Subject: Re: Unable to create a New VM - Error message: Unable
to
>> > start
>> > > > > instance due to Unable to get answer that is of class
>> > > > > com.cloud.agent.api.StartAnswer
>> > > > >
>> > > > > Hi Glenn,
>> > > > >
>> > > > > We have 16 hosts, all KVM.
>> > > > >
>> > > > > The error is not specific to Host 5. Every time I try to create
a
>> VM
>> > it
>> > > > > gives me the same error message but on different host.
>> > > > >
>> > > > > How do I restart the agent on the host. Is it service affecting?
>> > > > >
>> > > > > Thanks very much for your help.
>> > > > > Neza
>> > > > >
>> > > > > On 15 March 2016 at 17:37, Glenn Wagner <
>> glenn.wagner@shapeblue.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Can you explain your environment first , how many hosts
, what
>> is
>> > > your
>> > > > > > hypervisors (KVM,Xenserver,VMware)
>> > > > > >
>> > > > > > The error seems to be with your Host 5 Hypervisor
>> > > > > > Resource [Host:5] is unreachable:
>> > > > > >
>> > > > > > Go to the infrastructure section and see what host5 is
>> reporting,
>> > > > > > Have you tried to restart the agent on the Host?
>> > > > > >
>> > > > > > Thanks
>> > > > > > Glenn
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
>> > > > > > Glenn Wagner
>> > > > > > Senior Consultant ,  ShapeBlue
>> > > > > > d:  * | s: +27 21 527 0091*
>> <%7C%20s:%20+27%2021%20527%200091>  |
>> > > m:
>> > > > > > *+27 73 917 4111* <+27%2073%20917%204111>
>> > > > > > e:  *glenn.wagner@shapeblue.com | t: *
>> > > > > > <glenn.wagner@shapeblue.com%20%7C%20t:>  |  w:  *
>> www.shapeblue.com
>> > *
>> > > > > > <http://www.shapeblue.com>
>> > > > > > a:
>> > > > > > 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape
Town
>> > 7130
>> > > > > South Africa
>> > > > > > Shape Blue Ltd is a company incorporated in England &
Wales.
>> > > ShapeBlue
>> > > > > > Services India LLP is a company incorporated in India and
is
>> > operated
>> > > > > under
>> > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria
Ltda
>> is
>> > a
>> > > > > > company incorporated in Brasil and is operated under license
>> from
>> > > Shape
>> > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
The
>> > > Republic
>> > > > of
>> > > > > > South Africa and is traded under license from Shape Blue
Ltd.
>> > > ShapeBlue
>> > > > > is
>> > > > > > a registered trademark.
>> > > > > > This email and any attachments to it may be confidential
and are
>> > > > intended
>> > > > > > solely for the use of the individual to whom it is addressed.
>> Any
>> > > views
>> > > > > or
>> > > > > > opinions expressed are solely those of the author and do
not
>> > > > necessarily
>> > > > > > represent those of Shape Blue Ltd or related companies.
If you
>> are
>> > > not
>> > > > > the
>> > > > > > intended recipient of this email, you must neither take
any
>> action
>> > > > based
>> > > > > > upon its contents, nor copy or show it to anyone. Please
contact
>> > the
>> > > > > sender
>> > > > > > if you believe you have received this email in error.
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Nezar Madbouh [mailto:nezar.madbouh@ipscape.com.au]
>> > > > > > Sent: Tuesday, 15 March 2016 5:45 AM
>> > > > > > To: users@cloudstack.apache.org
>> > > > > > Subject: Re: Unable to create a New VM - Error message:
Unable
>> to
>> > > start
>> > > > > > instance due to Unable to get answer that is of class
>> > > > > > com.cloud.agent.api.StartAnswer
>> > > > > >
>> > > > > > Hi Jeremy,
>> > > > > >
>> > > > > > I have uploaded the contents to http://pastebin.com/1J3mmgTh
>> > > > > >
>> > > > > > your help is much appreciated,
>> > > > > > Nezar.
>> > > > > >
>> > > > > > On 15 March 2016 at 13:46, Jeremy Peterson <
>> jpeterson@acentek.net>
>> > > > > wrote:
>> > > > > >
>> > > > > > > I don't see any attachments can you create a pastebin.
That's
>> > > easier
>> > > > > > > to look through as I think attachments are removed
before
>> being
>> > > sent
>> > > > > > > to user group.
>> > > > > > >
>> > > > > > > Jeremy
>> > > > > > >
>> > > > > > >
>> > > > > > > -------- Original message --------
>> > > > > > > From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
>> > > > > > > Date: 3/14/2016 6:38 PM (GMT-06:00)
>> > > > > > > To: users@cloudstack.apache.org
>> > > > > > > Subject: Unable to create a New VM - Error message:
Unable to
>> > start
>> > > > > > > instance due to Unable to get answer that is of class
>> > > > > > > com.cloud.agent.api.StartAnswer
>> > > > > > >
>> > > > > > > Hi Shweta and Sadhu,
>> > > > > > >
>> > > > > > > Please find attached managerment-server.log which contains
the
>> > logs
>> > > > > > > since I started creating a new VM (VM id 600).
>> > > > > > >
>> > > > > > > Please note that we don't have agent.log files
>> > > > > > >
>> > > > > > > Your help is much appreciated.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Nezar.
>> > > > > > >
>> > > > > > Find out more about ShapeBlue and our range of CloudStack
>> related
>> > > > > services:
>> > > > > > IaaS Cloud Design & Build
>> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//>
| CSForge
>> –
>> > > rapid
>> > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
>> > > > > > CloudStack Consulting <
>> > http://shapeblue.com/cloudstack-consultancy/>
>> > > |
>> > > > > CloudStack
>> > > > > > Software Engineering
>> > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
>> > > > > > CloudStack Infrastructure Support
>> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/>
|
>> > > CloudStack
>> > > > > > Bootcamp Training Courses <
>> > http://shapeblue.com/cloudstack-training/
>> > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > DISCLAIMER
>> > ==========
>> > This e-mail may contain privileged and confidential information which is
>> > the property of Accelerite, a Persistent Systems business. It is
>> intended
>> > only for the use of the individual or entity to which it is addressed.
>> If
>> > you are not the intended recipient, you are not authorized to read,
>> retain,
>> > copy, print, distribute or use this message. If you have received this
>> > communication in error, please notify the sender and delete all copies
>> of
>> > this message. Accelerite, a Persistent Systems business does not accept
>> any
>> > liability for virus infected mails.
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message