cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nezar Madbouh <nezar.madb...@ipscape.com.au>
Subject Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Date Tue, 19 Apr 2016 02:25:19 GMT
Hi Ilya,

Adrian Sender provided us some support today and he managed to find the
problem. The problem is with the compute and disk offerings. We created new
compute offering and created a new VM from it and it worked.

Thank you very much for your help.

Regards,
Nezar.


On 16 April 2016 at 03:32, ilya <ilya.mailing.lists@gmail.com> wrote:

> Here is what i need to understand, if you are using SharedMountPoint,
> then /mnt/storage must be preset on all KVM hosts in a persistent manner
> (i.e. fstab).
>
> However, if you are using NFS, you can just let cloudstack manage the
> NFS mount points by adding the storage as NFS type.
>
> On 4/15/16 12:46 AM, ilya wrote:
> > Can you goto primary storage in cloudstack UI, is this what you have
> > defined for your NFS Server?
> >
> > "SharedMountPoint","host":"localhost","path":"/mnt/storage"
> >
> > If so, please use IP or FQDN instead..
> >
> >
> > On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> >> Hi Ilya,
> >>
> >> Here is the output of mount command:
> >>
> >> [root@eqx-cs-cmp-01 ~]# mount
> >> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> >> proc on /proc type proc (rw)
> >> sysfs on /sys type sysfs (rw)
> >> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> >> tmpfs on /dev/shm type tmpfs (rw)
> >> /dev/sda1 on /boot type ext4 (rw)
> >> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> >> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> >> nfsd on /proc/fs/nfsd type nfsd (rw)
> >> 10.61.154.10:/Cloudstack/primary on
> >> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >> [root@eqx-cs-cmp-01 ~]#
> >>
> >>
> >> On 15 April 2016 at 15:22, Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >> wrote:
> >>
> >>> Hi Ilya,
> >>>
> >>> Thanks for the instructions. I followed them and pasted the
> information on
> >>> pastebin:
> >>>
> >>> management-server logs job-14938
> >>> http://pastebin.com/GAiy8u5H
> >>>
> >>> Agent logs
> >>> http://pastebin.com/mfiADsJc
> >>>
> >>>
> >>> Your help is much appreciated.
> >>>
> >>> Nezar.
> >>>
> >>> On 15 April 2016 at 13:58, ilya <ilya.mailing.lists@gmail.com> wrote:
> >>>
> >>>> Here is how i would troubleshoot this:
> >>>>
> >>>> 1) ssh to MS host, run
> >>>>
> >>>> tail -f /var/cloudstack/management/management-server.log | grep -i
> >>>> StartVMCmdByAdmin
> >>>>
> >>>> 2) on cloudstack UI, press start on the VM and watch the tail log
> >>>>
> >>>> 3) on the tail log, you should see a StartVirtualMachine api call show
> >>>> up with job ID. For example:
> >>>>
> >>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
> >>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
> >>>> instanceId: 7648, cmd:
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> >>>>
> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
> >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
> >>>> null, lastPolled: null, created: null}
> >>>>
> >>>> Each task in cloudstack spins up its own job, in this case its
> >>>> *job-291575*
> >>>>
> >>>> 4) now execute
> >>>>> grep -A3 -B3 job-291575
> >>>> /var/cloudstack/management/management-server.log | tee job-291575.log
> >>>>> grep job-291575 /var/cloudstack/management/management-server.log |
> grep
> >>>> via | head -1
> >>>>
> >>>> you should see a response like so,
> >>>>
> >>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
> >>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576
> ctx-0988152e) Seq
> >>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138:  {
> Cmd ,
> >>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
> >>>> Flags: 100011,
> >>>>
> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
> >>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
> >>>>
> >>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
> >>>>
> >>>> SSH to my-kvm-host01.example.com, and copy the
> >>>> /var/log/cloudstack/agent.log to pastebin
> >>>>
> >>>> also copy job-291575.log to pastebin
> >>>>
> >>>> I'll try to have a look, though when i get chance.
> >>>>
> >>>>
> >>>> Also paste the output of mount command on kvm host in question. I'm
> >>>> suspecting one of your storage pools maybe bad.
> >>>>
> >>>> Lastly, what distro are you running for KVM?
> >>>>
> >>>> Regards
> >>>> ilya
> >>>>
> >>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
> >>>>> Hi Ilya,
> >>>>>
> >>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
> >>>>> Virtual router to be deleted automatically because no VM is running
> on
> >>>> this
> >>>>> vlan but it wasn't deleted.
> >>>>>
> >>>>> Am I right to expect the VR to be deleted automatically if no VM is
> >>>> running
> >>>>> in that vlan?
> >>>>>
> >>>>> Thanks,
> >>>>> Nezar.
> >>>>>
> >>>>> On 15 April 2016 at 09:53, Nezar Madbouh <
> nezar.madbouh@ipscape.com.au>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Ilya,
> >>>>>>
> >>>>>> I have run the script and it didn't return any results:
> >>>>>>
> >>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> >>>>>> [root@eqx-cs-mgt-01 ~]#
> >>>>>>
> >>>>>>
> >>>>>> I also ran the command inside mysql:
> >>>>>>
> >>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> >>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals  FROM
> >>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>>>>> "pseudoJobDispatcher";
> >>>>>> Empty set (0.00 sec)
> >>>>>>
> >>>>>> your help is much appreciated,
> >>>>>> Nezar.
> >>>>>>
> >>>>>>
> >>>>>> On 14 April 2016 at 16:49, ilya <ilya.mailing.lists@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> It appears you have pending jobs that are not letting you go
> through.
> >>>>>>>
> >>>>>>> Usually stop and starting management service should be enough.
> >>>>>>>
> >>>>>>> However, try running this script from your management server to
> show
> >>>> us
> >>>>>>> what you have running in async_job table.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> #!/bin/bash
> >>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
> >>>>>>>> DBPASS=$(java -classpath
> >>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
> >>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
> >>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
> >>>> awk
> >>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
> >>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> >>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> >>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.username | awk -F'=' '{print $2}')
> >>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
> >>>> db.cloud.name
> >>>>>>> | awk -F'=' '{print $2}')
> >>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.port | awk -F'=' '{print $2}')
> >>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> >>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
> >>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals  FROM
> >>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> The following work was done but unfortunately the problem is not
> >>>> fixed:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on management server:
> >>>>>>>>
> >>>>>>>> service cloudstack-management stop
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on the host:
> >>>>>>>>
> >>>>>>>> service cloudstack-agent stop
> >>>>>>>>
> >>>>>>>> service libvirtd restart
> >>>>>>>>
> >>>>>>>> service cloudstack-agent start
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on management server:
> >>>>>>>>
> >>>>>>>> service cloudstack-management start
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Nezar.
> >>>>>>>>
> >>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sweller@ena.com> wrote:
> >>>>>>>>
> >>>>>>>>> Yes, I think that order is fine. Make sure the processes are
> indeed
> >>>>>>>>> shutdown before moving on to the next step.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
> >>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>> start
> >>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>
> >>>>>>>>> Hi Simon,
> >>>>>>>>>
> >>>>>>>>> Can you please check if the following sequence of action is
> correct:
> >>>>>>>>>
> >>>>>>>>> on management server:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-management stop
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> on the host:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-agent stop
> >>>>>>>>>
> >>>>>>>>> service libvirtd restart
> >>>>>>>>>
> >>>>>>>>> service cloudstack-agent start
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> on management server:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-management start
> >>>>>>>>>
> >>>>>>>>> Thanks very much for your help.
> >>>>>>>>> Nezar.
> >>>>>>>>>
> >>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sweller@ena.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Shutting down all the management services should clear the jobs.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>
> >>>>>>>>>> Hi Simon,
> >>>>>>>>>>
> >>>>>>>>>> I will plan to restart agents, libvirtd and management services.
> >>>>>>>>>>
> >>>>>>>>>> How do I clear hung jobs.
> >>>>>>>>>>
> >>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>> Nezar.
> >>>>>>>>>>
> >>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sweller@ena.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Nezar,
> >>>>>>>>>>>
> >>>>>>>>>>> I've looked through the code related to your release and it
> seems
> >>>>>>> that
> >>>>>>>>> in
> >>>>>>>>>>> order to throw this exception, 10 attempts have been made to
> >>>> retrieve
> >>>>>>>>> the
> >>>>>>>>>>> disk information prior to the exception being thrown.
> >>>>>>>>>>>
> >>>>>>>>>>> It appears there is something really odd going on here.  I'm
> >>>> starting
> >>>>>>>>> to
> >>>>>>>>>>> wonder whether it might be worth while for you to stop all of
> your
> >>>>>>>>>> agents,
> >>>>>>>>>>> plus management server processes, restart libvirtd on each of
> the
> >>>>>>> hosts
> >>>>>>>>>> and
> >>>>>>>>>>> then bring the agents back up prior to starting the management
> >>>>>>> servers.
> >>>>>>>>>>>
> >>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
> >>>>>>>>> indicate
> >>>>>>>>>>> you have some jobs hung, for example:
> >>>>>>>>>>>
> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
> >>>> 1660851
> >>>>>>>>>> seconds
> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
> >>>> 4253003
> >>>>>>>>>> seconds
> >>>>>>>>>>>
> >>>>>>>>>>> Restarting the  agents, libvirtd and management will not cause
> an
> >>>>>>>>> outage
> >>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this,
> you
> >>>>>>>>> should
> >>>>>>>>>>> complete it in a maintenance window.
> >>>>>>>>>>>
> >>>>>>>>>>> - Si
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>
> >>>>>>>>>>> Please see answers to your questions below. I am not sure if I
> >>>>>>> provided
> >>>>>>>>>> the
> >>>>>>>>>>> correct answer for the primary and secondary storage
> >>>> configuration.
> >>>>>>>>>> Please
> >>>>>>>>>>> let me know if this is not what you are after:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>>>>> configuration
> >>>>>>>>>>> and your primary storage configuration from the UI with the
> >>>> details?
> >>>>>>>>>>>
> >>>>>>>>>>> mysql> select * from configuration where name like
> '%secondary%'\G
> >>>>>>>>>>> *************************** 1. row ***************************
> >>>>>>>>>>>      category: Account Defaults
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: management-server
> >>>>>>>>>>>          name: max.account.secondary.storage
> >>>>>>>>>>>         value: 400
> >>>>>>>>>>>   description: The default maximum secondary storage space (in
> >>>> GiB)
> >>>>>>>>> that
> >>>>>>>>>>> can be used for an account
> >>>>>>>>>>> default_value: 400
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> *************************** 2. row ***************************
> >>>>>>>>>>>      category: Project Defaults
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: management-server
> >>>>>>>>>>>          name: max.project.secondary.storage
> >>>>>>>>>>>         value: 400
> >>>>>>>>>>>   description: The default maximum secondary storage space (in
> >>>> GiB)
> >>>>>>>>> that
> >>>>>>>>>>> can be used for a project
> >>>>>>>>>>> default_value: 400
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> *************************** 3. row ***************************
> >>>>>>>>>>>      category: Hidden
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: management-server
> >>>>>>>>>>>          name: secondary.storage.vm
> >>>>>>>>>>>         value: +rwRhJOXPymbQOuAMLnhiw==
> >>>>>>>>>>>   description: Deploys a VM per zone to manage secondary
> storage
> >>>> if
> >>>>>>>>> true,
> >>>>>>>>>>> otherwise secondary storage is mounted on management server
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> *************************** 4. row ***************************
> >>>>>>>>>>>      category: Advanced
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: ExtensionRegistry
> >>>>>>>>>>>          name: secondary.storage.vm.allocators.exclude
> >>>>>>>>>>>         value: NULL
> >>>>>>>>>>>   description: Extensions to exclude from being registered
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>>       updated: 2014-12-04 00:18:15
> >>>>>>>>>>>         scope: Global
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> *************************** 5. row ***************************
> >>>>>>>>>>>      category: NetworkManager
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: management-server
> >>>>>>>>>>>          name: vm.network.nic.max.secondary.ipaddresses
> >>>>>>>>>>>         value: NULL
> >>>>>>>>>>>   description: Specify the number of secondary ip addresses per
> >>>> nic
> >>>>>>> per
> >>>>>>>>>> vm
> >>>>>>>>>>> default_value: 256
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> 5 rows in set (0.00 sec)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> mysql> select * from configuration where name like
> '%primary%'\G
> >>>>>>>>>>> *************************** 1. row ***************************
> >>>>>>>>>>>      category: Account Defaults
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: management-server
> >>>>>>>>>>>          name: max.account.primary.storage
> >>>>>>>>>>>         value: 200
> >>>>>>>>>>>   description: The default maximum primary storage space (in
> GiB)
> >>>>>>> that
> >>>>>>>>>> can
> >>>>>>>>>>> be used for an account
> >>>>>>>>>>> default_value: 200
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> *************************** 2. row ***************************
> >>>>>>>>>>>      category: Project Defaults
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: management-server
> >>>>>>>>>>>          name: max.project.primary.storage
> >>>>>>>>>>>         value: 200
> >>>>>>>>>>>   description: The default maximum primary storage space (in
> GiB)
> >>>>>>> that
> >>>>>>>>>> can
> >>>>>>>>>>> be used for a project
> >>>>>>>>>>> default_value: 200
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> *************************** 3. row ***************************
> >>>>>>>>>>>      category: Storage
> >>>>>>>>>>>      instance: DEFAULT
> >>>>>>>>>>>     component: TemplateManager
> >>>>>>>>>>>          name: primary.storage.download.wait
> >>>>>>>>>>>         value: 10800
> >>>>>>>>>>>   description: In second, timeout for download template to
> primary
> >>>>>>>>>> storage
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>>       updated: NULL
> >>>>>>>>>>>         scope: NULL
> >>>>>>>>>>>    is_dynamic: 0
> >>>>>>>>>>> 3 rows in set (0.00 sec)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Also, what is the Cloudstack version you are running and what
> is
> >>>> the
> >>>>>>> OS
> >>>>>>>>>>> you're running it on?
> >>>>>>>>>>>
> >>>>>>>>>>> Cloudstack 4.4.1
> >>>>>>>>>>>
> >>>>>>>>>>> Centos 6.6
> >>>>>>>>>>>
> >>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>>>>
> >>>>>>>>>>> Primary storage on Hypervisors has the following parameters
> >>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
> >>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >>>>>>>>>>>
> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Secondary storage on SSVM has the following parameters
> >>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
> >>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
> >>>>>>> 2147483647
> >>>>>>>
> >>>>
> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>> Nezar.
> >>>>>>>>>>>
> >>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sweller@ena.com>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Also, what is the Cloudstack version you are running and what
> is
> >>>> the
> >>>>>>>>> OS
> >>>>>>>>>>>> you're running it on?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>>>>>
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Simon Weller <sweller@ena.com>
> >>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nexar,
> >>>>>>>>>>>>
> >>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
> >>>> the
> >>>>>>>>> host
> >>>>>>>>>>>> where the VM resides.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>>>>>> configuration
> >>>>>>>>>>>> and your primary storage configuration from the UI with the
> >>>> details?
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Si
> >>>>>>>>>>>>
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the
> SSVM:
> >>>>>>>>>>>>
> >>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p
> 3922
> >>>>>>>>>>>> root@169.254.3.161
> >>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I can only ssh from one Host:
> >>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p
> 3922
> >>>>>>>>>>>> root@169.254.3.161
> >>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
> >>>>>>>>> nezar.madbouh@ipscape.com.au
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was
> able
> >>>> to
> >>>>>>>>>>> create
> >>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
> >>>> environment.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sweller@ena.com>
> >>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm assuming you have other VMs successfully running,
> correct?
> >>>> So
> >>>>>>>>>> you
> >>>>>>>>>>>> can
> >>>>>>>>>>>>>> write to your primary storage.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
> >>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Simon and Shweta,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Please see the result below:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> First DNS server is  8.8.8.8
> >>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> >>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
> >>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
> >>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> >>>>>>>>>>>>>> Good: Can ping DNS server
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> nfs is currently mounted
> >>>>>>>>>>>>>> Mount point is
> >>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> >>>>>>>>>>>>>> Good: Can write to mount point
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
> >>>>>>>>>>>>>> Good: Can connect to management server port 8250
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Good: Java process is running
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
> >>>>>>>>>>>>>> root@s-11-VM:~#
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sweller@ena.com>
> >>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Log into the Cloudstack UI.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on  View
> All
> >>>>>>>>>> below
> >>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM"
> and
> >>>>>>>>>> then
> >>>>>>>>>>>>>> click
> >>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
> >>>>>>>>> You'll
> >>>>>>>>>>> see
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and
> also
> >>>>>>>>> the
> >>>>>>>>>>>> Link
> >>>>>>>>>>>>>>> Local IP Address line and then follow the original
> >>>> instructions
> >>>>>>>>> in
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>> previous email.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>> From: Shweta Agarwal <shweta.agarwal1@accelerite.com>
> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Nezar,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
> >>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
> >>>> host
> >>>>>>>>>>>> details
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> system vms.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> Shweta
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sweller@ena.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
> >>>> the
> >>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the
> host
> >>>>>>>>> the
> >>>>>>>>>>> SSVM
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
> >>>>>>>>> /root/.ssh/id_rsa.cloud
> >>>>>>>>>>> -p
> >>>>>>>>>>>>>> 3922
> >>>>>>>>>>>>>>>> root@link-local-ip  (replace link-local-ip with the link
> >>>>>>>>> local
> >>>>>>>>>>> ip).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Try  running the SSVM diangnostic script:
> >>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
> >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>> Unable
> >>>>>>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Please see details below:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
> >>>>>>>>> where
> >>>>>>>>>>> id =
> >>>>>>>>>>>>>> 5;
> >>>>>>>>>>>>>>>> Reading table information for completion of table and
> column
> >>>>>>>>>> names
> >>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup
> with
> >>>> -A
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Database changed
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> | id | name                         | status | type    |
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
> >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't
> find
> >>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>
> >>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> >>>>>>>>>>>>>>>>         at com.cloud.utils.nio.Task.run(Task.java:84)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>>>>>>>>>>>         at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
> >>>>>>>>>>>>>>>> Name                 State      Autostart
> >>>>>>>>>>>>>>>> -----------------------------------------
> >>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active     no
> >>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active     no
> >>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
> >>>>>>>>>>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
> >>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
> >>>>>>>>>>>>>>>>                       203G 215G 1.3G 1% /
> >>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
> >>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
> >>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
> >>>>>>>>>>>>>>>>                       7.6T  4.1T  3.5T  54%
> >>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain
> with
> >>>>>>>>>>>> matching
> >>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>> 'i-11-600-VM'
> >>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found:
> no
> >>>>>>>>>>> storage
> >>>>>>>>>>>>>> vol
> >>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sweller@ena.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
> >>>>>>>>>> prior
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> throwing the host exception.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into
> your
> >>>>>>>>>>> MySQL
> >>>>>>>>>>>>>>> server
> >>>>>>>>>>>>>>>>> and run this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where
> id=5;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can you post the results?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If it's a host, can you post the
> >>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> that host?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
> >>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>> Unable
> >>>>>>>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
> >>>>>>>>> network",
> >>>>>>>>>> do
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>> mean
> >>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
> >>>>>>>>>> can't
> >>>>>>>>>>>> ping
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server
> and
> >>>>>>>>>>> Host
> >>>>>>>>>>>>>> and I
> >>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I
> launched
> >>>>>>>>> the
> >>>>>>>>>>>>>> console
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> couldn't login.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Please note that we need to take into consideration that
> the
> >>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
> >>>>>>>>>>> service
> >>>>>>>>>>>>>>> outage
> >>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> much as possible.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sweller@ena.com
> >
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
> >>>>>>>>> and
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> management server can't communicate with it.
> >>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network
> where
> >>>>>>>>>> you
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> attempting to start an instance.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
> >>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>>> Unable
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Glenn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
> >>>>>>>>>>> create
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> VM
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
> >>>>>>>>>>> affecting?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks very much for your help.
> >>>>>>>>>>>>>>>>>> Neza
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
> >>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
> >>>>>>>>> ,
> >>>>>>>>>>> what
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
> >>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
> >>>>>>>>>>>>>> reporting,
> >>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>> Glenn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
> >>>>>>>>>>>>>>>>>>> Glenn Wagner
> >>>>>>>>>>>>>>>>>>> Senior Consultant ,  ShapeBlue
> >>>>>>>>>>>>>>>>>>> d:  * | s: +27 21 527 0091*
> >>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091>  |
> >>>>>>>>>>>>>>>> m:
> >>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
> >>>>>>>>>>>>>>>>>>> e:  *glenn.wagner@shapeblue.com | t: *
> >>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:>  |  w:  *
> >>>>>>>>>>>>>> www.shapeblue.com
> >>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
> >>>>>>>>>>>>>>>>>>> a:
> >>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> >>>>>>>>>> Cape
> >>>>>>>>>>>> Town
> >>>>>>>>>>>>>>> 7130
> >>>>>>>>>>>>>>>>>> South Africa
> >>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
> >>>>>>>>>> Wales.
> >>>>>>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
> >>>>>>>>> and
> >>>>>>>>>> is
> >>>>>>>>>>>>>>> operated
> >>>>>>>>>>>>>>>>>> under
> >>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
> >>>>>>>>> Consultoria
> >>>>>>>>>>>> Ltda
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
> >>>>>>>>>> license
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> Shape
> >>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >>>>>>>>> by
> >>>>>>>>>>> The
> >>>>>>>>>>>>>>>> Republic
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape
> Blue
> >>>>>>>>>>> Ltd.
> >>>>>>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> a registered trademark.
> >>>>>>>>>>>>>>>>>>> This email and any attachments to it may be
> confidential
> >>>>>>>>>> and
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>>> Any
> >>>>>>>>>>>>>>>> views
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and
> do
> >>>>>>>>>> not
> >>>>>>>>>>>>>>>>> necessarily
> >>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
> >>>>>>>>> If
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
> >>>>>>>>>> any
> >>>>>>>>>>>>>> action
> >>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone.
> Please
> >>>>>>>>>>>> contact
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> sender
> >>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
> >>>>>>>>> nezar.madbouh@ipscape.com.au]
> >>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
> >>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>>>> Unable
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Jeremy,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have uploaded the contents to
> >>>>>>>>>>> http://pastebin.com/1J3mmgTh
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> your help is much appreciated,
> >>>>>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
> >>>>>>>>>>>>>> jpeterson@acentek.net>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
> >>>>>>>>>>>> That's
> >>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
> >>>>>>>>>> before
> >>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>> sent
> >>>>>>>>>>>>>>>>>>>> to user group.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jeremy
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -------- Original message --------
> >>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madbouh@ipscape.com.au>
> >>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
> >>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
> >>>>>>>>>> Unable
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
> >>>>>>>>>> contains
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> logs
> >>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
> >>>>>>>>> CloudStack
> >>>>>>>>>>>>>> related
> >>>>>>>>>>>>>>>>>> services:
> >>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
> >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
> >>>>>>>>>>>> CSForge
> >>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>> rapid
> >>>>>>>>>>>>>>>>>>> IaaS deployment framework <
> >>>>>>>>> http://shapeblue.com/csforge/>
> >>>>>>>>>>>>>>>>>>> CloudStack Consulting <
> >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
> >>>>>>>>>>>>>>>> |
> >>>>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>> Software Engineering
> >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/
> >
> >>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
> >>>>>>>>>> |
> >>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
> >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DISCLAIMER
> >>>>>>>>>>>>>>> ==========
> >>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
> >>>> information
> >>>>>>>>>>> which
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business.
> It
> >>>> is
> >>>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized
> to
> >>>>>>>>>> read,
> >>>>>>>>>>>>>> retain,
> >>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
> >>>>>>>>> received
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> communication in error, please notify the sender and delete
> >>>> all
> >>>>>>>>>>> copies
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business
> does
> >>>> not
> >>>>>>>>>>>> accept
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>> liability for virus infected mails.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message