Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of
 pdion@cloudops.com)
MIME-Version: 1.0
Date: Thu, 16 Jan 2014 17:16:35 -0500
Message-ID: 
 <CAD97ioeW46q-2W7tqzdbmOwGnyeeYxgg7iR_EvPS43REmaV4NA@mail.gmail.com>
Subject: Issues after upgrading CS from 4.1 to 4.2.1 managing XenServer 6.0.2
 clusters
From: Pierre-Luc Dion <pdion@cloudops.com>
To: users@cloudstack.apache.org
Content-Type: multipart/mixed; boundary=089e0112c1ba5977d804f01dc7fa

--089e0112c1ba5977d804f01dc7fa
Content-Type: multipart/alternative; boundary=089e0112c1ba5977d404f01dc7f8

--089e0112c1ba5977d404f01dc7f8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

We've upgrade our Cloudstack environment from 4.1 to 4.2.1, it is currently
managing XenServer 6.0.2 clusters.

1. Since we have upgraded to 4.2.1 we noticed that Secondary Storage Name
have been rename and we now see UUID's of SS.
2. On one cluster that is using local-Storage and Shared Storage we have
removed on host from the XenServer Pool, since then,  this error keep
repeating in the management-server.log:

2014-01-16 17:01:17,898 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-437:null) Seq 1-590413869: Response Received:
2014-01-16 17:01:17,898 DEBUG [agent.transport.Request]
(StatsCollector-2:null) Seq 1-590413869: Received:  { Ans: , MgmtId:
152067623530867, via: 1, Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2014-01-16 17:01:17,949 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-73:null) Seq 2-1599340596: Executing request
2014-01-16 17:01:18,191 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-320:null) Ping from 69
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 0.006406249999999999
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 12.9175
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 24.08
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 0.29500000000000004
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 23.8075
2014-01-16 17:01:18,370 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 10.09875
2014-01-16 17:01:18,370 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 0.29828125
2014-01-16 17:01:18,370 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 2.6500000000000004
2014-01-16 17:01:18,370 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Vm cpu utilization 0.8375
2014-01-16 17:01:18,507 WARN  [xen.resource.CitrixResourceBase]
(DirectAgent-73:null) Error while collecting disk stats from :
You gave an invalid object reference.  The object may have recently been
deleted.  The class parameter gives the type of reference given, and the
handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnecti=
on.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VBDMetrics.getIoReadKbs(VBDMetrics.java:210)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVmStats(CitrixResou=
rceBase.java:2791)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResource=
Base.java:2691)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixR=
esourceBase.java:497)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenSer=
ver56Resource.java:59)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java=
:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access=
$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc=
heduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1=
146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:=
615)
at java.lang.Thread.run(Thread.java:701)
2014-01-16 17:01:18,509 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-73:null) Seq 2-1599340596: Response Received:


Also, Because we were planning to upgrade the pool using Local-Storage we
start moving VM into another Primary Storage using regular procedure of
turning off Instances, and moving them as Admin account. this keep failing
but also delete Instance DISK which result of having data lost.

Now, we can't see our hosts list and System VM list from the Infrastructure
tab (see attachements), we only see ERROR.

here is the apilog.log when trying to show hosts:

2014-01-16 17:13:35,337 INFO  [cloud.api.ApiServer] (catalina-exec-18:null)
(userId=3D2 accountId=3D2 sessionId=3D4F907F8E77BE76ADF46BB4FBE6B23BC0)
172.24.0.20 -- GET
command=3DlistHosts&response=3Djson&sessionkey=3DtPdf1uSuogiM1ubmRFuPuFV%2F=
LzM%3D&page=3D1&pageSize=3D20&type=3Drouting&listAll=3Dtrue&_=3D13899104152=
01
530 null

is their any other logs to look at to understands what's broken because the
upper stacktrace is clueless for me and their is no errors in the
management-server.log when trying to display hosts.

Regards,

Pierre-Luc Dion
Architecte de Solution Cloud | Cloud Solutions Architect
514-447-3456, 1101
- - -

*CloudOps*420 rue Guy
Montr=E9al QC  H3J 1S6
www.cloudops.com
@CloudOps_

--089e0112c1ba5977d404f01dc7f8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">We&#39;ve upgrade our Cloudstack environment from 4.1 to 4=
.2.1, it is currently managing XenServer 6.0.2 clusters.=A0<div><br></div><=
div>1. Since we have upgraded to 4.2.1 we noticed that Secondary Storage Na=
me have been rename and we now see UUID&#39;s of SS.</div>
<div>2. On one cluster that is using local-Storage and Shared Storage we ha=
ve removed on host from the XenServer Pool, since then, =A0this error keep =
repeating in the management-server.log:</div><div><br></div><font size=3D"1=
">2014-01-16 17:01:17,898 DEBUG [agent.manager.DirectAgentAttache] (DirectA=
gent-437:null) Seq 1-590413869: Response Received: <br>
2014-01-16 17:01:17,898 DEBUG [agent.transport.Request] (StatsCollector-2:n=
ull) Seq 1-590413869: Received: =A0{ Ans: , MgmtId: 152067623530867, via: 1=
, Ver: v1, Flags: 10, { GetVmStatsAnswer } }<br>2014-01-16 17:01:17,949 DEB=
UG [agent.manager.DirectAgentAttache] (DirectAgent-73:null) Seq 2-159934059=
6: Executing request<br>
2014-01-16 17:01:18,191 DEBUG [agent.manager.DirectAgentAttache] (DirectAge=
nt-320:null) Ping from 69<br>2014-01-16 17:01:18,369 DEBUG [xen.resource.Ci=
trixResourceBase] (DirectAgent-73:null) Vm cpu utilization 0.00640624999999=
9999<br>
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase] (DirectAgen=
t-73:null) Vm cpu utilization 12.9175<br>2014-01-16 17:01:18,369 DEBUG [xen=
.resource.CitrixResourceBase] (DirectAgent-73:null) Vm cpu utilization 24.0=
8<br>
2014-01-16 17:01:18,369 DEBUG [xen.resource.CitrixResourceBase] (DirectAgen=
t-73:null) Vm cpu utilization 0.29500000000000004<br>2014-01-16 17:01:18,36=
9 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-73:null) Vm cpu util=
ization 23.8075<br>
2014-01-16 17:01:18,370 DEBUG [xen.resource.CitrixResourceBase] (DirectAgen=
t-73:null) Vm cpu utilization 10.09875<br>2014-01-16 17:01:18,370 DEBUG [xe=
n.resource.CitrixResourceBase] (DirectAgent-73:null) Vm cpu utilization 0.2=
9828125<br>
2014-01-16 17:01:18,370 DEBUG [xen.resource.CitrixResourceBase] (DirectAgen=
t-73:null) Vm cpu utilization 2.6500000000000004<br>2014-01-16 17:01:18,370=
 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-73:null) Vm cpu utili=
zation 0.8375<br>
2014-01-16 17:01:18,507 WARN =A0[xen.resource.CitrixResourceBase] (DirectAg=
ent-73:null) Error while collecting disk stats from : <br>You gave an inval=
id object reference. =A0The object may have recently been deleted. =A0The c=
lass parameter gives the type of reference given, and the handle parameter =
echoes the bad value given.<br>
at com.xensource.xenapi.Types.checkResponse(Types.java:209)<br>at com.xenso=
urce.xenapi.Connection.dispatch(Connection.java:368)<br>at com.cloud.hyperv=
isor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenS=
erverConnectionPool.java:909)<br>
at com.xensource.xenapi.VBDMetrics.getIoReadKbs(VBDMetrics.java:210)<br>at =
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVmStats(CitrixResou=
rceBase.java:2791)<br>at com.cloud.hypervisor.xen.resource.CitrixResourceBa=
se.execute(CitrixResourceBase.java:2691)<br>
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(Citr=
ixResourceBase.java:497)<br>at com.cloud.hypervisor.xen.resource.XenServer5=
6Resource.executeRequest(XenServer56Resource.java:59)<br>at com.cloud.agent=
.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)<br>
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<=
br>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)<br=
>at java.util.concurrent.FutureTask.run(FutureTask.java:166)<br>at java.uti=
l.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Sch=
eduledThreadPoolExecutor.java:165)<br>
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run=
(ScheduledThreadPoolExecutor.java:266)<br>at java.util.concurrent.ThreadPoo=
lExecutor.runWorker(ThreadPoolExecutor.java:1146)<br>at java.util.concurren=
t.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
at java.lang.Thread.run(Thread.java:701)<br>2014-01-16 17:01:18,509 DEBUG [=
agent.manager.DirectAgentAttache] (DirectAgent-73:null) Seq 2-1599340596: R=
esponse Received:</font><div><div><div dir=3D"ltr"><div><br></div><div><br>
</div><div>Also, Because we were planning to upgrade the pool using Local-S=
torage we start moving VM into another Primary Storage using regular proced=
ure of turning off Instances, and moving them as Admin account. this keep f=
ailing but also delete Instance DISK which result of having data lost.=A0</=
div>
<div><br></div><div>Now, we can&#39;t see our hosts list and System VM list=
 from the Infrastructure tab (see attachements), we only see ERROR.</div><d=
iv>


<p class=3D"">here is the apilog.log when trying to show hosts:</p><p class=
=3D""><font size=3D"1">2014-01-16 17:13:35,337 INFO=A0 [cloud.api.ApiServer=
] (catalina-exec-18:null) (userId=3D2 accountId=3D2 sessionId=3D4F907F8E77B=
E76ADF46BB4FBE6B23BC0) 172.24.0.20 -- GET command=3DlistHosts&amp;response=
=3Djson&amp;sessionkey=3DtPdf1uSuogiM1ubmRFuPuFV%2FLzM%3D&amp;page=3D1&amp;=
pageSize=3D20&amp;type=3Drouting&amp;listAll=3Dtrue&amp;_=3D1389910415201 5=
30 null</font></p>
</div><div><br></div><div>is their any other logs to look at to understands=
 what&#39;s broken because the upper stacktrace is clueless for me and thei=
r is no errors in the management-server.log when trying to display hosts.</=
div>
<div><br></div><div>Regards,</div><div><br></div>Pierre-Luc Dion<br><div><f=
ont face=3D"arial, helvetica, sans-serif"><span style=3D"line-height:18.187=
5px">Architecte de Solution Cloud | Cloud Solutions Architect</span><br></f=
ont></div>
514-447-3456, 1101<br><div>- - -<strong><br>CloudOps<br></strong>420 rue Gu=
y<br>Montr=E9al=A0QC =A0H3J 1S6<br><a href=3D"http://www.cloudops.com/" rel=
=3D"nofollow" target=3D"_blank">www.cloudops.com</a><br>@CloudOps_<br></div=
><div><br>
</div><div><br></div></div></div>
</div></div>

--089e0112c1ba5977d404f01dc7f8--

--089e0112c1ba5977d804f01dc7fa--