cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Young <iyo...@ratespecial.com>
Subject Re: failed to start virtual router
Date Wed, 30 Apr 2014 18:58:03 GMT
I think my problem stems from a partially downloaded system VM template.  I
just noticed systemvm-kvm-4.3 is stuck at 27% downloaded.  It must have
been interrupted during the upgrade to 4.3.  At the moment I've rolled back
to 4.2.1 with a somewhat usable management interface, although the system
VMs won't start.  I suspect there is something in the database that is
causing it to try to use the 4.3 template.  How can I delete the template
and make sure the management server is using the older one?


On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <iyoung@ratespecial.com> wrote:

> Ok, so I've figured out a way to identify volumes in the filesystem.  For
> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad is the
> root volume for an instance I want to back up.  Is this in qcow2 format or
> something else?  I'm using KVM.
>
>
> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <iyoung@ratespecial.com> wrote:
>
>> Now I can't start cloudstack-agent.  The agent.log says:
>>
>> Unable to start agent: Failed to get private nic name
>>
>> I know this is because the network bridge is no longer set up correctly.
>>  I used to have a cloud0 and a cloudbr0 interface.  Now I only have
>> cloudbr0.  I haven't changed my network configuration.  Somehow it's been
>> changed by CloudStack during the upgrade/downgrade.  This is getting worse
>> and worse the more I try to recover my data.  Is there any way to back up
>> the instances' volumes via the command line?  I can't tell which is which
>> because the filenames are all hashes.  I really need to get these instances
>> up and running--there are several months worth of work at stake here.
>>
>>
>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <breeze7086@gmail.com> wrote:
>>
>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1 safely?
>>>
>>>
>>> 2014-04-30 8:45 GMT+08:00 Ian Young <iyoung@ratespecial.com>:
>>>
>>> > Ok, my Cloudstack installation is now so broken that I think it's
>>> probably
>>> > best to backup all my instances and templates, wipe the databases, and
>>> > start from scratch.  However, I can't take snapshots or download
>>> volumes
>>> > anymore.  What's causing these errors?
>>> >
>>> > 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>>> > com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>> command,
>>> > due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>> Commands
>>> > 841744457 to Host 1 timed out after 21600
>>> > 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>>> > com.cloud.utils.exception.CloudRuntimeException:
>>> > com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>> command,
>>> > due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>> Commands
>>> > 841744457 to Host 1 timed out after 21600
>>> > 2014-04-29 17:40:51,269 WARN  [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data object
>>> > (VOLUME,
>>> > org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@7bbbd901),
>>> no
>>> > need to delete from object in store ref table
>>> > 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>> > (Job-Executor-11:ctx-0a3ead79) Unexpected exception while executing
>>> > org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>>> > com.cloud.utils.exception.CloudRuntimeException: Failed to copy the
>>> volume
>>> > from the source primary storage pool to secondary storage.
>>> > 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus:
>>> FAILED,
>>> > resultCode: 530, result:
>>> >
>>> >
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>> > to copy the volume from the source primary storage pool to secondary
>>> > storage."}
>>> >
>>> >
>>> > On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <iyoung@ratespecial.com>
>>> wrote:
>>> >
>>> > > I downgraded to 4.2.1 again but cloudstack-management won't start
>>> because
>>> > > the database is version 4.3. Is it safe to restore the database
>>> backup I
>>> > > made prior to this whole process? In the meantime I have destroyed
>>> and
>>> > > created system VMs, so I'm not sure it's a good idea.
>>> > > On Apr 29, 2014 3:09 PM, "Ian Young" <iyoung@ratespecial.com>
wrote:
>>> > >
>>> > >> @stevenliang: I take it back--you can't set the VM size when you
>>> > register
>>> > >> the template.
>>> > >>
>>> > >>
>>> > >> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <motty.cruz@gmail.com>
>>> > wrote:
>>> > >>
>>> > >>> yes, you would have to shutdown the router, then click on "Change
>>> > Service
>>> > >>> Offering"
>>> > >>> restart the VR.
>>> > >>>
>>> > >>> To Ian,
>>> > >>>
>>> > >>> I suspect you forgot the last step: " cloudstack-setup-management"
>>> > >>>
>>> > >>> that would fix your issue, I think,
>>> > >>>
>>> > >>> Thanks,
>>> > >>> ---
>>> > >>> I downgraded to 4.2.1 and then upgraded to 4.3.  Now the
>>> > >>> cloudstack-management service can't start because it can't
connect
>>> to
>>> > the
>>> > >>> database.
>>> > >>>
>>> > >>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null)
>>> Unable
>>> > >>> to
>>> > >>> get a new db connection
>>> > >>> Caused by: java.sql.SQLException: Access denied for user 'cloud'@
>>> > >>> 'localhost'
>>> > >>> (using password: YES)
>>> > >>>
>>> > >>> Where are the credentials stored?
>>> > >>>
>>> > >>>
>>> > >>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <
>>> stevenliang@yesup.com>
>>> > >>> wrote:
>>> > >>>
>>> > >>> > oh, then change service offering for vr?
>>> > >>> >
>>> > >>> >
>>> > >>> > On 29/04/14 05:53 PM, motty cruz wrote:
>>> > >>> >
>>> > >>> >> for my VR, I created a new
>>> > >>> >>
>>> > >>> >>   "System Offering For Software Router"
>>> > >>> >> CPU in (MHz) 1.00GHz
>>> > >>> >> Memory (in MB) 1.00GB
>>> > >>> >>
>>> > >>> >> this are my current offerings, I'm sure the more RAM
and CPU
>>> better
>>> > >>> >> performance.
>>> > >>> >>
>>> > >>> >> Thanks,
>>> > >>> >>
>>> > >>> >>
>>> > >>> >>
>>> > >>> >> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>>> stevenliang@yesup.com
>>> > >
>>> > >>> >> wrote:
>>> > >>> >>
>>> > >>> >>  Thank you again, motty.
>>> > >>> >>> I didn't notice this earlier.
>>> > >>> >>> BTW, how did you make your vr had 1GB CPU and
512MB RAM?
>>> > >>> >>>
>>> > >>> >>>
>>> > >>> >>>
>>> > >>> >>> On 29/04/14 05:33 PM, motty cruz wrote:
>>> > >>> >>>
>>> > >>> >>>  Stevellang,
>>> > >>> >>>> I not sure if you saw this in the forums earlier
:
>>> > >>> >>>>
>>> > >>>
>>> >
>>> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/%
>>> > >>> >>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>> > >>> c2rfEDRoY9mA@mail.gmail.com%3E
>>> > >>> >>>>
>>> > >>> >>>> I don't know if the bug was fixed yet,
>>> > >>> >>>>
>>> > >>> >>>> I will try upgrade in the next couple of days
on a testing
>>> > cluster,
>>> > >>> will
>>> > >>> >>>> report back if the bug was fixed.
>>> > >>> >>>>
>>> > >>> >>>> Thanks,
>>> > >>> >>>>
>>> > >>> >>>>
>>> > >>> >>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang
<
>>> > stevenliang@yesup.com
>>> > >>> >
>>> > >>> >>>> wrote:
>>> > >>> >>>>
>>> > >>> >>>>   Thank you, motty.
>>> > >>> >>>>
>>> > >>> >>>>> I am also running kvm. Since that time
I failed upgrade, I am
>>> > still
>>> > >>> >>>>> using
>>> > >>> >>>>> 4.2.1. I'll try as your advice.
>>> > >>> >>>>>
>>> > >>> >>>>>
>>> > >>> >>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>> > >>> >>>>>
>>> > >>> >>>>>   Stevenllang,
>>> > >>> >>>>>
>>> > >>> >>>>>> I had the similar issue with VR, I
notice it was because I
>>> leave
>>> > >>> the
>>> > >>> >>>>>> default system specs on the VR, for
instance by default
>>> 500MHz
>>> > on
>>> > >>> CPU
>>> > >>> >>>>>> and
>>> > >>> >>>>>> 128MB on RAM, if you upgrade to at
least 1GB on CPU and
>>> 512MB of
>>> > >>> RAM
>>> > >>> >>>>>> your
>>> > >>> >>>>>> VR will survive the upgrade from 4.2.1
to 4.3.1.
>>> > >>> >>>>>>
>>> > >>> >>>>>> I am running KVM, when I upgrade from
4.2.1 to 4.3 my VMs
>>> were
>>> > not
>>> > >>> >>>>>> able
>>> > >>> >>>>>> to
>>> > >>> >>>>>> access outside world, even if I created
a new router.
>>> > >>> >>>>>>
>>> > >>> >>>>>> wish you the best,
>>> > >>> >>>>>> -motty
>>> > >>> >>>>>>
>>> > >>> >>>>>>
>>> > >>> >>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang
<
>>> > >>> stevenliang@yesup.com>
>>> > >>> >>>>>> wrote:
>>> > >>> >>>>>>
>>> > >>> >>>>>>    Yes, I had two zones(one is basic,
another is advanced
>>> mode).
>>> > >>> >>>>>>
>>> > >>> >>>>>>  After I upgraded from 4.2.1 to 4.3,
the vrouter lost.
>>> > >>> >>>>>>> So I rolled back to 4.2.1, the
vrouter came back.
>>> > >>> >>>>>>>
>>> > >>> >>>>>>>
>>> > >>> >>>>>>> On 29/04/14 04:54 PM, Ian Young
wrote:
>>> > >>> >>>>>>>
>>> > >>> >>>>>>>    Did rolling back to 4.2 fix
the problem?
>>> > >>> >>>>>>>
>>> > >>> >>>>>>>  On Tue, Apr 29, 2014 at 1:22
PM, stevenliang <
>>> > >>> stevenliang@yesup.com
>>> > >>> >>>>>>>> >
>>> > >>> >>>>>>>> wrote:
>>> > >>> >>>>>>>>
>>> > >>> >>>>>>>>     I met your situation before.
Finally I rolled back to
>>> 4.2
>>> > >>> >>>>>>>>
>>> > >>> >>>>>>>>   On 29/04/14 04:18 PM, Ian
Young wrote:
>>> > >>> >>>>>>>>
>>> > >>> >>>>>>>>>     I destroyed the old
virtual router and was able to
>>> create
>>> > >>> a new
>>> > >>> >>>>>>>>> one
>>> > >>> >>>>>>>>> by
>>> > >>> >>>>>>>>>
>>> > >>> >>>>>>>>>   adding a new instance.
 However, this new router also
>>> > failed
>>> > >>> to
>>> > >>> >>>>>>>>>
>>> > >>> >>>>>>>>>> start,
>>> > >>> >>>>>>>>>> citing the same error.
 After that, the expungement
>>> delay
>>> > >>> elapsed
>>> > >>> >>>>>>>>>> and
>>> > >>> >>>>>>>>>> the
>>> > >>> >>>>>>>>>> virtual router was
expunged, so now I have none.
>>> > >>> >>>>>>>>>>
>>> > >>> >>>>>>>>>>
>>> > >>> >>>>>>>>>> On Mon, Apr 28, 2014
at 8:52 PM, Ian Young <
>>> > >>> >>>>>>>>>> iyoung@ratespecial.com>
>>> > >>> >>>>>>>>>> wrote:
>>> > >>> >>>>>>>>>>
>>> > >>> >>>>>>>>>>      I upgraded from
4.2.1 to 4.3.0 tonight, following
>>> the
>>> > >>> >>>>>>>>>> instructions
>>> > >>> >>>>>>>>>> here:
>>> > >>> >>>>>>>>>>
>>> > >>> >>>>>>>>>>
>>> http://docs.cloudstack.apache.org/projects/cloudstack-
>>> > >>> >>>>>>>>>>
>>> > >>> >>>>>>>>>>
>>> >  release-notes/en/latest/rnotes.html#upgrade-from-4-2-x-to-4-3
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> At the last step,
I tried to restart the system VMs.
>>>  The
>>> > >>> virtual
>>> > >>> >>>>>>>>>>> router
>>> > >>> >>>>>>>>>>> failed to start.
 Here is the message that was
>>> displayed in
>>> > >>> the
>>> > >>> >>>>>>>>>>> web
>>> > >>> >>>>>>>>>>> UI:
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> Resource [Host:1]
is unreachable: Host 1: Unable to
>>> start
>>> > >>> >>>>>>>>>>> instance
>>> > >>> >>>>>>>>>>> due
>>> > >>> >>>>>>>>>>> to
>>> > >>> >>>>>>>>>>> Unable to start
VM[DomainRouter|r-4-VM] due to error in
>>> > >>> >>>>>>>>>>> finalizeStart,
>>> > >>> >>>>>>>>>>> not
>>> > >>> >>>>>>>>>>> retrying
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> I tried running
the script to restart the VMs but this
>>> time
>>> > >>> it
>>> > >>> >>>>>>>>>>> failed
>>> > >>> >>>>>>>>>>> to
>>> > >>> >>>>>>>>>>> start the console
proxy:
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> [root@virthost1
~]$ cloudstack-sysvmadm -d
>>> 192.168.100.6
>>> > -u
>>> > >>> >>>>>>>>>>> cloud
>>> > >>> >>>>>>>>>>> -p
>>> > >>> >>>>>>>>>>> -a
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> Stopping and starting
1 secondary storage vm(s)...
>>> > >>> >>>>>>>>>>> Done stopping
and starting secondary storage vm(s)
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> Stopping and starting
1 console proxy vm(s)...
>>> > >>> >>>>>>>>>>> ERROR: Failed
to start console proxy vm with id 2
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> Done stopping
and starting console proxy vm(s) .
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> Stopping and starting
0 running routing vm(s)...
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>> Is there a way
to wipe the system VMs out and start
>>> over?
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>>
>>> > >>> >>>>>>>>>>>
>>> > >>> >
>>> > >>>
>>> > >>
>>> > >>
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message