Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
From: Jeremy Peterson <jpeterson@acentek.net>
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: RE: Recreating SystemVM's
Thread-Topic: Recreating SystemVM's
Thread-Index: AdLebC9IxTUgENEqTbaEWv4l7WAdCQASNPWAAAwVVQAABUGrEAAENCIAABw9PIAAEOJMQAAf0yUA//+3Rlv//eyzIA==
Date: Fri, 9 Jun 2017 13:30:13 +0000
Message-ID: <df431bd6cba24830a36b8f6b056fdeb0@ACEGROUP-EXCH.acegroup.local>
References: <4af89340c32e4e7ba956bb31dc30ea0e@ACEGROUP-EXCH.acegroup.local>
 <CAP997Ue8iSmoRDzf_3HmTXEF=wueFw+8A+V4-piC7rTUvsTgRg@mail.gmail.com>
 <bee3dc81-ce64-4afe-9556-9c23d15a1678@msgid.missiveapp.com>
 <6323085176904cc09a33a8ab8081a311@ACEGROUP-EXCH.acegroup.local>
 <907ebb26981e4220af72faaed43dab75@ACEGROUP-EXCH.acegroup.local>
 <CABdC_7kqAx4OBHvErStK1X2yuVgRRvnYe2CySQcc+yNGuBDdSA@mail.gmail.com>
 <a1f42ebffa1540bda6a6a0716adaf24b@ACEGROUP-EXCH.acegroup.local>,<CAPpf5a0teUJb1Etk5t5rQn=1sbDXGPs5jn69T3AMcyNKKWoJ3Q@mail.gmail.com>
 <4030d67b515a451dbee8b41cd3888df7@ACEGROUP-EXCH.acegroup.local>
In-Reply-To: <4030d67b515a451dbee8b41cd3888df7@ACEGROUP-EXCH.acegroup.local>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
archived-at: Fri, 09 Jun 2017 13:30:29 -0000

08:28:43	select * from vm_instance where name like 's-%' limit 10000	7481 r=
ow(s) returned	0.000 sec / 0.032 sec

All vm's 'state' returned Destoryed outside of the current vm 7873 which is=
 in a Stopped state but that goes Destroyed and a new get created.

Any other suggestions?

Jeremy


-----Original Message-----
From: Jeremy Peterson [mailto:jpeterson@acentek.net]=20
Sent: Thursday, June 8, 2017 12:47 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

I'll make that change in the am.

Today I put a host in maintence and rebooted because proxy and secstore vm =
were constantly being created on that host and still no change.

Let you know tomorrow.

Jeremy


Sent from my Verizon, Samsung Galaxy smartphone


-------- Original message --------
From: Rajani Karuturi <rajani@apache.org>
Date: 6/8/17 12:07 AM (GMT-06:00)
To: Users <users@cloudstack.apache.org>
Subject: Re: Recreating SystemVM's

Did you check SMLog on xenserver?
unable to destroy task(com.xensource.xenapi.Task@256829a8) on
host(b34f086e-fabf-471e-9feb-8f54362d7d0f) due to You gave an invalid objec=
t reference.  The object may have recently been deleted.  The class paramet=
er gives the type of reference given, and the handle parameter echoes the b=
ad value given.

Looks like Destroy of SSVM failed. What state is SSVM in? mark it as Destro=
yed in cloud DB and wait for cloudstack to create a new SSVM.

~Rajani
http://cloudplatform.accelerite.com/

On Thu, Jun 8, 2017 at 1:11 AM, Jeremy Peterson <jpeterson@acentek.net>
wrote:

> Probably agreed.
>
> But I ran toolstack restart on all hypervisors and v-3193 just tried=20
> to create and fail along with s-5398.
>
> The PIF error went away. But VM's are still recreating
>
> https://pastebin.com/4n4xBgMT
>
> New log from this afternoon.
>
> My catalina.out is over 4GB
>
> Jeremy
>
>
> -----Original Message-----
> From: Makrand [mailto:makrandsanap@gmail.com]
> Sent: Wednesday, June 7, 2017 12:52 AM
> To: users@cloudstack.apache.org
> Subject: Re: Recreating SystemVM's
>
> Hi there,
>
> Looks more like hypervisor issue.
>
> Just run *xe-toolstack-restart* on hosts where these VMs are trying to=20
> start or if you don't have too many hosts, better run on all members=20
> including master. most of i/o related issues squared off by toolstack=20
> bounce.
>
> --
> Makrand
>
>
> On Wed, Jun 7, 2017 at 3:01 AM, Jeremy Peterson=20
> <jpeterson@acentek.net>
> wrote:
>
> > Ok so I pulled this from Sunday morning.
> >
> > https://pastebin.com/nCETw1sC
> >
> >
> > errorInfo: [HOST_CANNOT_ATTACH_NETWORK,=20
> > OpaqueRef:65d0c844-bd70-81e9-4518-8809e1dc0ee7,
> > OpaqueRef:0093ac3f-9f3a-37e1-9cdb-581398d27ba2]
> >
> > XenServer error.
> >
> > Now this still gets me because all of the other VM's launched just fine=
.
> >
> > Going into XenCenter I see an error at the bottom This PIF is a bond=20
> > slave and cannot be plugged.
> >
> > ???
> >
> > If I go to networking on the hosts I see the storage vlans and bonds=20
> > are all there.
> >
> > I see my GUEST-PUB bond is there and LACP is setup correct.
> >
> > Any suggestions ?
> >
> >
> > Jeremy
> >
> >
> > -----Original Message-----
> > From: Jeremy Peterson [mailto:jpeterson@acentek.net]
> > Sent: Tuesday, June 6, 2017 9:23 AM
> > To: users@cloudstack.apache.org
> > Subject: RE: Recreating SystemVM's
> >
> > Thank you all for those responses.
> >
> > I'll comb through my management-server.log and post a pastebin if=20
> > I'm scratching my head.
> >
> > Jeremy
> >
> > -----Original Message-----
> > From: Rajani Karuturi [mailto:rajani@apache.org]
> > Sent: Tuesday, June 6, 2017 6:53 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: Recreating SystemVM's
> >
> > If the zone is enabled, cloudstack should recreate them automatically.
> >
> > ~ Rajani
> >
> > http://cloudplatform.accelerite.com/
> >
> > On June 6, 2017 at 11:37 AM, Erik Weber (terbolous@gmail.com)
> > wrote:
> >
> > CloudStack should recreate automatically, check the mgmt server logs=20
> > for hints of why it doesn't happen.
> >
> > --
> > Erik
> >
> > tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson
> > <jpeterson@acentek.net>:
> >
> > I had an issue Sunday morning with cloudstack 4.9.0 and xenserver 6.5.0=
.
> > My hosts stop sending LACP PDU's and caused a network drop to iSCSI=20
> > primary storage.
> >
> > So all my instances recovered via HA enabled.
> >
> > But my console proxy and secondary storage system VM's got stuck in=20
> > a boot state that would not power on.
> >
> > At this time they are expunged and gone.
> >
> > How do I tell cloudstack-management to recreate system VM's?
> >
> > I'm drawing a blank since deploying CS two years ago and just=20
> > keeping things running and adding hosts and more storage everything=20
> > has been so stable.
> >
> > Jeremy
> >
>