Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of iyoung@ratespecial.com
 designates 209.85.223.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAO0CpcQ+dKKvWbyYHCSRcQ1nd+Vh7RiAHkZkBE0_Oq_nBcFbbw@mail.gmail.com>
References: 
 <CAO0CpcTM1DeFP2ikk+JTysLk87S9fsKoRNO+yonJqh4byPcJ5w@mail.gmail.com>
	<CAO0CpcTC-ua-FKC44QYZaNfaVFe7txcdB0JSj++gNSi5J870GQ@mail.gmail.com>
	<53600A03.3090304@yesup.com>
	<CAO0CpcR4HzniA869AKji+mhQkE8CxbCFfiy0ZtUSzWYbmY=8RQ@mail.gmail.com>
	<53601612.4070001@yesup.com>
	<CALoOYy5HUOyMnXVsnJDhrq7U14PL+2DyD3OZNaFYPDd8wPpjfA@mail.gmail.com>
	<536018D3.1050601@yesup.com>
	<CALoOYy51iSMGjur=TpsqOF6A+KQVDZXXak40GQEmQy0_A7mE8w@mail.gmail.com>
	<53601D48.6080404@yesup.com>
	<CALoOYy5MyxbsTO8Uj08c9qxQKcm2abVuBkKNMvA-=4WHsZvPOQ@mail.gmail.com>
	<53602058.6020908@yesup.com>
	<CALoOYy7qeSNxx370rGDH-rVmxnAGicaEssBBg0J2LoL-TVm3bg@mail.gmail.com>
	<CAO0CpcTRjonaCTfS6ibHbRbS_=Rbh3JC476imfy3qPXEoTOw1A@mail.gmail.com>
	<CAO0CpcQwCtmAKMDPu8D4-kge6mE_VeCRGBT+3ZAB6+PVM+P2+g@mail.gmail.com>
	<CAO0CpcSh7VsZSjqLkh4mbsSCUgsGOzY1idaM5O+pd=RRH9z4ow@mail.gmail.com>
	<CAEo=yRP6iGN__ukFBwFo9RrPmXHHTJAA9KVptCiG6ZxA9SgWhg@mail.gmail.com>
	<CAO0CpcQ+dKKvWbyYHCSRcQ1nd+Vh7RiAHkZkBE0_Oq_nBcFbbw@mail.gmail.com>
Date: Tue, 29 Apr 2014 20:23:22 -0700
Message-ID: 
 <CAO0CpcRHXq2cUoFCmRtznjU433LkfwnJrq3hjyzuV5zgFwLxGg@mail.gmail.com>
Subject: Re: failed to start virtual router
From: Ian Young <iyoung@ratespecial.com>
To: users@cloudstack.apache.org
Content-Type: multipart/alternative; boundary=047d7b1119012060f704f83a12da

--047d7b1119012060f704f83a12da
Content-Type: text/plain; charset=UTF-8

Ok, so I've figured out a way to identify volumes in the filesystem.  For
instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad is the
root volume for an instance I want to back up.  Is this in qcow2 format or
something else?  I'm using KVM.


On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <iyoung@ratespecial.com> wrote:

> Now I can't start cloudstack-agent.  The agent.log says:
>
> Unable to start agent: Failed to get private nic name
>
> I know this is because the network bridge is no longer set up correctly.
>  I used to have a cloud0 and a cloudbr0 interface.  Now I only have
> cloudbr0.  I haven't changed my network configuration.  Somehow it's been
> changed by CloudStack during the upgrade/downgrade.  This is getting worse
> and worse the more I try to recover my data.  Is there any way to back up
> the instances' volumes via the command line?  I can't tell which is which
> because the filenames are all hashes.  I really need to get these instances
> up and running--there are several months worth of work at stake here.
>
>
> On Tue, Apr 29, 2014 at 6:13 PM, ma y <breeze7086@gmail.com> wrote:
>
>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1 safely?
>>
>>
>> 2014-04-30 8:45 GMT+08:00 Ian Young <iyoung@ratespecial.com>:
>>
>> > Ok, my Cloudstack installation is now so broken that I think it's
>> probably
>> > best to backup all my instances and templates, wipe the databases, and
>> > start from scratch.  However, I can't take snapshots or download volumes
>> > anymore.  What's causing these errors?
>> >
>> > 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>> > com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>> > due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands
>> > 841744457 to Host 1 timed out after 21600
>> > 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>> > com.cloud.utils.exception.CloudRuntimeException:
>> > com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>> > due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands
>> > 841744457 to Host 1 timed out after 21600
>> > 2014-04-29 17:40:51,269 WARN  [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data object
>> > (VOLUME,
>> > org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@7bbbd901),
>> no
>> > need to delete from object in store ref table
>> > 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> > (Job-Executor-11:ctx-0a3ead79) Unexpected exception while executing
>> > org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>> > com.cloud.utils.exception.CloudRuntimeException: Failed to copy the
>> volume
>> > from the source primary storage pool to secondary storage.
>> > 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus:
>> FAILED,
>> > resultCode: 530, result:
>> >
>> >
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>> > to copy the volume from the source primary storage pool to secondary
>> > storage."}
>> >
>> >
>> > On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <iyoung@ratespecial.com>
>> wrote:
>> >
>> > > I downgraded to 4.2.1 again but cloudstack-management won't start
>> because
>> > > the database is version 4.3. Is it safe to restore the database
>> backup I
>> > > made prior to this whole process? In the meantime I have destroyed and
>> > > created system VMs, so I'm not sure it's a good idea.
>> > > On Apr 29, 2014 3:09 PM, "Ian Young" <iyoung@ratespecial.com> wrote:
>> > >
>> > >> @stevenliang: I take it back--you can't set the VM size when you
>> > register
>> > >> the template.
>> > >>
>> > >>
>> > >> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <motty.cruz@gmail.com>
>> > wrote:
>> > >>
>> > >>> yes, you would have to shutdown the router, then click on "Change
>> > Service
>> > >>> Offering"
>> > >>> restart the VR.
>> > >>>
>> > >>> To Ian,
>> > >>>
>> > >>> I suspect you forgot the last step: " cloudstack-setup-management"
>> > >>>
>> > >>> that would fix your issue, I think,
>> > >>>
>> > >>> Thanks,
>> > >>> ---
>> > >>> I downgraded to 4.2.1 and then upgraded to 4.3.  Now the
>> > >>> cloudstack-management service can't start because it can't connect
>> to
>> > the
>> > >>> database.
>> > >>>
>> > >>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null)
>> Unable
>> > >>> to
>> > >>> get a new db connection
>> > >>> Caused by: java.sql.SQLException: Access denied for user 'cloud'@
>> > >>> 'localhost'
>> > >>> (using password: YES)
>> > >>>
>> > >>> Where are the credentials stored?
>> > >>>
>> > >>>
>> > >>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <stevenliang@yesup.com
>> >
>> > >>> wrote:
>> > >>>
>> > >>> > oh, then change service offering for vr?
>> > >>> >
>> > >>> >
>> > >>> > On 29/04/14 05:53 PM, motty cruz wrote:
>> > >>> >
>> > >>> >> for my VR, I created a new
>> > >>> >>
>> > >>> >>   "System Offering For Software Router"
>> > >>> >> CPU in (MHz) 1.00GHz
>> > >>> >> Memory (in MB) 1.00GB
>> > >>> >>
>> > >>> >> this are my current offerings, I'm sure the more RAM and CPU
>> better
>> > >>> >> performance.
>> > >>> >>
>> > >>> >> Thanks,
>> > >>> >>
>> > >>> >>
>> > >>> >>
>> > >>> >> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>> stevenliang@yesup.com
>> > >
>> > >>> >> wrote:
>> > >>> >>
>> > >>> >>  Thank you again, motty.
>> > >>> >>> I didn't notice this earlier.
>> > >>> >>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>> > >>> >>>
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> On 29/04/14 05:33 PM, motty cruz wrote:
>> > >>> >>>
>> > >>> >>>  Stevellang,
>> > >>> >>>> I not sure if you saw this in the forums earlier :
>> > >>> >>>>
>> > >>>
>> > http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/%
>> > >>> >>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>> > >>> c2rfEDRoY9mA@mail.gmail.com%3E
>> > >>> >>>>
>> > >>> >>>> I don't know if the bug was fixed yet,
>> > >>> >>>>
>> > >>> >>>> I will try upgrade in the next couple of days on a testing
>> > cluster,
>> > >>> will
>> > >>> >>>> report back if the bug was fixed.
>> > >>> >>>>
>> > >>> >>>> Thanks,
>> > >>> >>>>
>> > >>> >>>>
>> > >>> >>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <
>> > stevenliang@yesup.com
>> > >>> >
>> > >>> >>>> wrote:
>> > >>> >>>>
>> > >>> >>>>   Thank you, motty.
>> > >>> >>>>
>> > >>> >>>>> I am also running kvm. Since that time I failed upgrade, I am
>> > still
>> > >>> >>>>> using
>> > >>> >>>>> 4.2.1. I'll try as your advice.
>> > >>> >>>>>
>> > >>> >>>>>
>> > >>> >>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>> > >>> >>>>>
>> > >>> >>>>>   Stevenllang,
>> > >>> >>>>>
>> > >>> >>>>>> I had the similar issue with VR, I notice it was because I
>> leave
>> > >>> the
>> > >>> >>>>>> default system specs on the VR, for instance by default
>> 500MHz
>> > on
>> > >>> CPU
>> > >>> >>>>>> and
>> > >>> >>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and
>> 512MB of
>> > >>> RAM
>> > >>> >>>>>> your
>> > >>> >>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>> > >>> >>>>>>
>> > >>> >>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my VMs
>> were
>> > not
>> > >>> >>>>>> able
>> > >>> >>>>>> to
>> > >>> >>>>>> access outside world, even if I created a new router.
>> > >>> >>>>>>
>> > >>> >>>>>> wish you the best,
>> > >>> >>>>>> -motty
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>> > >>> stevenliang@yesup.com>
>> > >>> >>>>>> wrote:
>> > >>> >>>>>>
>> > >>> >>>>>>    Yes, I had two zones(one is basic, another is advanced
>> mode).
>> > >>> >>>>>>
>> > >>> >>>>>>  After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>> > >>> >>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>> > >>> >>>>>>>
>> > >>> >>>>>>>
>> > >>> >>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>> > >>> >>>>>>>
>> > >>> >>>>>>>    Did rolling back to 4.2 fix the problem?
>> > >>> >>>>>>>
>> > >>> >>>>>>>  On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>> > >>> stevenliang@yesup.com
>> > >>> >>>>>>>> >
>> > >>> >>>>>>>> wrote:
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>     I met your situation before. Finally I rolled back to
>> 4.2
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>   On 29/04/14 04:18 PM, Ian Young wrote:
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>>     I destroyed the old virtual router and was able to
>> create
>> > >>> a new
>> > >>> >>>>>>>>> one
>> > >>> >>>>>>>>> by
>> > >>> >>>>>>>>>
>> > >>> >>>>>>>>>   adding a new instance.  However, this new router also
>> > failed
>> > >>> to
>> > >>> >>>>>>>>>
>> > >>> >>>>>>>>>> start,
>> > >>> >>>>>>>>>> citing the same error.  After that, the expungement delay
>> > >>> elapsed
>> > >>> >>>>>>>>>> and
>> > >>> >>>>>>>>>> the
>> > >>> >>>>>>>>>> virtual router was expunged, so now I have none.
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>> > >>> >>>>>>>>>> iyoung@ratespecial.com>
>> > >>> >>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>>      I upgraded from 4.2.1 to 4.3.0 tonight, following
>> the
>> > >>> >>>>>>>>>> instructions
>> > >>> >>>>>>>>>> here:
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>>
>> http://docs.cloudstack.apache.org/projects/cloudstack-
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>>
>> >  release-notes/en/latest/rnotes.html#upgrade-from-4-2-x-to-4-3
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> At the last step, I tried to restart the system VMs.
>>  The
>> > >>> virtual
>> > >>> >>>>>>>>>>> router
>> > >>> >>>>>>>>>>> failed to start.  Here is the message that was
>> displayed in
>> > >>> the
>> > >>> >>>>>>>>>>> web
>> > >>> >>>>>>>>>>> UI:
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to
>> start
>> > >>> >>>>>>>>>>> instance
>> > >>> >>>>>>>>>>> due
>> > >>> >>>>>>>>>>> to
>> > >>> >>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to error in
>> > >>> >>>>>>>>>>> finalizeStart,
>> > >>> >>>>>>>>>>> not
>> > >>> >>>>>>>>>>> retrying
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> I tried running the script to restart the VMs but this
>> time
>> > >>> it
>> > >>> >>>>>>>>>>> failed
>> > >>> >>>>>>>>>>> to
>> > >>> >>>>>>>>>>> start the console proxy:
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d
>> 192.168.100.6
>> > -u
>> > >>> >>>>>>>>>>> cloud
>> > >>> >>>>>>>>>>> -p
>> > >>> >>>>>>>>>>> -a
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)...
>> > >>> >>>>>>>>>>> Done stopping and starting secondary storage vm(s)
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>> > >>> >>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> Is there a way to wipe the system VMs out and start
>> over?
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>>
>> > >>> >
>> > >>>
>> > >>
>> > >>
>> >
>>
>
>

--047d7b1119012060f704f83a12da--