Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Wed, 28 Aug 2013 09:36:51 +0000 (UTC)
From: "Harikrishna Patnala (JIRA)" <jira@apache.org>
To: cloudstack-issues@incubator.apache.org
Message-ID: <JIRA.12665470.1377506439010.45983.1377682611624@arcas>
In-Reply-To: <JIRA.12665470.1377506439010@arcas>
References: <JIRA.12665470.1377506439010@arcas>
Subject: [jira] [Commented] (CLOUDSTACK-4499) Xen6.1/Xen6.2 hosts initially
 transition to 'Alert' and then to 'Up' after addHost
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CLOUDSTACK-4499?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=
=3D13752241#comment-13752241 ]=20

Harikrishna Patnala commented on CLOUDSTACK-4499:
-------------------------------------------------

2nd issue is because there is some delay in xenserver bridge to be active, =
in mean time we are trying to the plug the VIF which is causing the problem=
.
               =20
> Xen6.1/Xen6.2 hosts initially transition to 'Alert' and then to 'Up' afte=
r addHost
> -------------------------------------------------------------------------=
---------
>
>                 Key: CLOUDSTACK-4499
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-449=
9
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the defa=
ult.)=20
>          Components: API, Management Server, XenServer
>    Affects Versions: 4.2.0
>            Reporter: Prasanna Santhanam
>            Priority: Critical
>             Fix For: 4.2.1
>
>         Attachments: cloud.sql.tar.bz2, mslog.tar.bz2, SMlog
>
>
> The same bug was reported in CLOUDSTACK-3839 which has been determined to=
 be a
> UI issue but the problem is with the addHost API and Xenserver hosts 6.1/=
6.2.=20
> When a freshly provisioned 6.1/6.2 host is added to cloudstack it initial=
ly
> goes into 'Alert' state thereby failing the storage pool creation subsequ=
ently.
> StoragePool addition requires the hosts to be in Up state. But a minute o=
r two
> later the Xenserver host automatically moves back to Up state after which
> storage pool addition can occur
> When the host is added to cloudstack:
> 2013-08-26 06:46:22,965 DEBUG [agent.manager.DirectAgentAttache] (DirectA=
gent-1:null) Seq 1-1916403719: Executing request
> 2013-08-26 06:46:23,261 DEBUG [xen.resource.CitrixResourceBase] (DirectAg=
ent-1:null) Can't find a vif on dom0 for link local, creating a new one
> 2013-08-26 06:46:23,274 DEBUG [xen.resource.CitrixResourceBase] (DirectAg=
ent-1:null) Lowest available Vif device number: 0 for VM: Control domain on=
 host: apache-81-3
> 2013-08-26 06:46:23,600 WARN  [xen.resource.CitrixResourceBase] (DirectAg=
ent-1:null) Unable to create local link network
> The server failed to handle your request, due to an internal error.  The =
given message may give details useful for debugging the problem.
>     at com.xensource.xenapi.Types.checkResponse(Types.java:1694)
>     at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
>     at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServe=
rConnection.dispatch(XenServerConnectionPool.java:909)
>     at com.xensource.xenapi.VIF.plug(VIF.java:846)
> Host marked Alert:
> 2013-08-26 06:46:23,616 DEBUG [agent.manager.AgentManagerImpl] (catalina-=
exec-14:null) Sending Disconnect to listener: com.cloud.network.NetworkUsag=
eManagerImpl$DirectNetworkStatsListener
> 2013-08-26 06:46:23,616 DEBUG [cloud.network.NetworkUsageManagerImpl] (ca=
talina-exec-14:null) Disconnected called on 1 with status Alert
> 2013-08-26 06:46:23,616 DEBUG [agent.manager.AgentManagerImpl] (catalina-=
exec-14:null) Sending Disconnect to listener: com.cloud.consoleproxy.Consol=
eProxyListener
> 2013-08-26 06:46:23,619 DEBUG [cloud.host.Status] (catalina-exec-14:null)=
 Transition:[Resource state =3D Enabled, Agent event =3D AgentDisconnected,=
 Host id =3D 1, name =3D apache-81-3]
> 2013-08-26 06:46:23,625 DEBUG [cloud.host.Status] (catalina-exec-14:null)=
 Agent status update: [id =3D 1; name =3D apache-81-3; old status =3D Conne=
cting; event =3D AgentDisconnected; new status =3D Alert; old update count =
=3D 1; new update count =3D 2]
> Automatic retry by CloudStack a few moments later:
> 2013-08-26 06:46:52,752 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManag=
er Timer:null) Completed acquiring hosts for clusters not owned by any mana=
gement server
> 2013-08-26 06:46:52,756 DEBUG [agent.manager.ClusteredAgentManagerImpl] (=
ClusteredAgentManager Timer:null) Found 2 unmanaged direct hosts, processin=
g connect for them...
> 2013-08-26 06:46:52,757 DEBUG [agent.manager.ClusteredAgentManagerImpl] (=
ClusteredAgentManager Timer:null) Loading directly connected host 1(apache-=
81-3)
> Meanwhile Storage Pool addition fails:
> 2013-08-26 06:46:52,766 DEBUG [cloud.api.ApiServlet] (catalina-exec-17:nu=
ll) =3D=3D=3DEND=3D=3D=3D  10.208.8.5 -- GET  username=3Droot&apiKey=3DsLUj=
l2n1c33JJpY4ZzMtObUgp9Ah2xW5inofCclZQQ1V6xp5k3CmAsGhemLwQxVvQY67EiopM_fbWBo=
JRKCIww&podid=3Dfdf30680-d5c7-425e-bd7f-fb8c52e7cb96&hypervisor=3DXenServer=
&clusterid=3D10ff9a88-54bc-405b-9b4b-c26acf9caca8&zoneid=3D0431023e-ff78-43=
b6-a32d-262f36a78cbb&command=3DaddHost&url=3Dhttp%3A%2F%2Fapache-81-2&signa=
ture=3DoJqOno9flCtPMSSHH%2FZD%2BJl5QXw%3D&response=3Djson
> 2013-08-26 06:46:52,773 DEBUG [cloud.api.ApiServlet] (catalina-exec-16:nu=
ll) =3D=3D=3DSTART=3D=3D=3D  10.208.8.5 -- GET  apiKey=3DsLUjl2n1c33JJpY4Zz=
MtObUgp9Ah2xW5inofCclZQQ1V6xp5k3CmAsGhemLwQxVvQY67EiopM_fbWBoJRKCIww&name=
=3Dz0p0c0ps0&url=3Dnfs%3A%2F%2Fnfs.fmt.vmops.com%3A%2Fexport%2Fautomation%2=
Facs%2Fprimary&podid=3Dfdf30680-d5c7-425e-bd7f-fb8c52e7cb96&clusterid=3D10f=
f9a88-54bc-405b-9b4b-c26acf9caca8&zoneid=3D0431023e-ff78-43b6-a32d-262f36a7=
8cbb&command=3DcreateStoragePool&signature=3D70CuFbmf58PU4X0QbF1tKbBGT60%3D=
&response=3Djson
> 2013-08-26 06:46:52,787 DEBUG [cloud.network.NetworkModelImpl] (Clustered=
AgentManager Timer:null) Failed to retrive the default label for storage tr=
affic:zone: 1 hypervisor: XenServer due to:Unable to find the default physi=
cal network with traffic=3DStorage in the specified zone id
> 2013-08-26 06:46:52,839 DEBUG [datastore.lifecycle.CloudStackPrimaryDataS=
toreLifeCycleImpl] (catalina-exec-16:null) createPool Params @ scheme - nfs=
 storageHost - nfs.fmt.vmops.com hostPath - /export/automation/acs/primary =
port - -1
> 2013-08-26 06:46:52,889 DEBUG [agent.manager.ClusteredAgentManagerImpl] (=
ClusteredAgentManager Timer:null) Loading directly connected host 2(apache-=
81-2)
> 2013-08-26 06:46:52,892 DEBUG [agent.manager.AgentManagerImpl] (AgentTask=
Pool-2:null) Simulating start for resource 192267e4-d97d-4e37-b899-82b157e3=
166f id 1
> 2013-08-26 06:46:52,900 DEBUG [cloud.storage.StorageManagerImpl] (catalin=
a-exec-16:null) Failed to add data store
> com.cloud.utils.exception.CloudRuntimeException: No host up to associate =
a storage pool with in cluster 1
>     at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimar=
yDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleIm=
pl.java:371)
>     at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl=
.java:749)
>     at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl=
.java:177)
> Host comes back to Up state:
> 2013-08-26 06:46:53,898 DEBUG [agent.manager.DirectAgentAttache] (DirectA=
gent-4:null) Seq 1-1519321095: Executing request
> 2013-08-26 06:46:54,279 DEBUG [xen.resource.CitrixResourceBase] (DirectAg=
ent-4:null) already have a vif on dom0 for link local network
> 2013-08-26 06:46:54,703 DEBUG [agent.transport.Request] (AgentTaskPool-2:=
null) Seq -1--1: Startup request from directly connected host:  { Cmd , Mgm=
tId: -1, via: -1, Ver: v1, Flags: 11, [{"cpus":4,"speed":2261,"memory":1619=
0149248,"dom0MinMemory":588251136,"poolSync":false,"_clusterVMStates":{},"c=
aps":"xen-3.0-x86_64 , xen-3.0-x86_32p , hvm-3.0-x86_32 , hvm-3.0-x86_32p ,=
 hvm-3.0-x86_64","pool":"9dfa936a-30f7-b5d1-4d4e-b5a75facb313","hypervisorT=
ype":"XenServer","hostDetails":{"product_version":"6.1.0","private.network.=
device":"Pool-wide network associated with eth0","com.cloud.network.Network=
s.RouterPrivateIpStrategy":"DcGlobal","product_brand":"XenServer","product_=
version_text_short":"6.1","can_bridge_firewall":"false"},"hypervisorVersion=
":"6.1.0","type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":=
"192267e4-d97d-4e37-b899-82b157e3166f","name":"apache-81-3","version":"4.2.=
0","iqn":"iqn.2013-08.com.vmops.fmt:691cf4d5","publicIpAddress":"10.208.8.6=
9","publicNetmask":"255.255.255.192","publicMacAddress":"b8:ac:6f:94:f1:67"=
,"privateIpAddress":"10.208.8.69","privateMacAddress":"b8:ac:6f:94:f1:67","=
privateNetmask":"255.255.255.192","storageIpAddress":"10.208.8.69","storage=
Netmask":"255.255.255.192","storageMacAddress":"b8:ac:6f:94:f1:67","wait":0=
},{"totalSize":0,"poolInfo":{"uuid":"ae248ed7-dd2a-cfc8-53f7-f43661c596c8",=
"host":"10.208.8.69","localPath":"lvm","hostPath":"lvm","poolType":"LVM","c=
apacityBytes":491505319936,"availableBytes":491501125632},"resourceType":"S=
TORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","guid":"192=
267e4-d97d-4e37-b899-82b157e3166f","wait":0}] }
> 2013-08-26 06:46:54,709 DEBUG [cloud.resource.ResourceManagerImpl] (Agent=
TaskPool-2:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT=
_CONNECT to BaremetalPxeManagerImpl_EnhancerByCloudStack_d79b93ae
> 2013-08-26 06:46:54,709 DEBUG [cloud.resource.ResourceManagerImpl] (Agent=
TaskPool-2:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT=
_CONNECT to PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_b51c5fd=
e
> 2013-08-26 06:46:54,709 DEBUG [cloud.resource.ResourceManagerImpl] (Agent=
TaskPool-2:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT=
_CONNECT to LxcServerDiscoverer_EnhancerByCloudStack_449ab3e6
> 2013-08-26 06:46:54,709 DEBUG [cloud.resource.ResourceManagerImpl] (Agent=
TaskPool-2:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT=
_CONNECT to NiciraNvp
> 2013-08-26 06:46:54,709 DEBUG [cloud.resource.ResourceManagerImpl] (Agent=
TaskPool-2:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT=
_CONNECT to OvmDiscoverer_EnhancerByCloudStack_ec7ad054
> 2013-08-26 06:46:54,710 DEBUG [cloud.resource.ResourceManagerImpl] (Agent=
TaskPool-2:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT=
_CONNECT to XcpServerDiscoverer_EnhancerByCloudStack_831158ad
> 2013-08-26 06:46:54,710 INFO  [xen.discoverer.XcpServerDiscoverer] (Agent=
TaskPool-2:null) Host: apache-81-3 connected with hypervisor type: XenServe=
r. Checking CIDR...
> 2013-08-26 06:46:54,735 DEBUG [cloud.resource.ResourceState] (AgentTaskPo=
ol-2:null) Resource state update: [id =3D 1; name =3D apache-81-3; old stat=
e =3D Enabled; event =3D InternalCreated; new state =3D Enabled]
> 2013-08-26 06:46:54,736 DEBUG [cloud.host.Status] (AgentTaskPool-2:null) =
Transition:[Resource state =3D Enabled, Agent event =3D AgentConnected, Hos=
t id =3D 1, name =3D apache-81-3]
> 2013-08-26 06:46:54,742 DEBUG [cloud.host.Status] (AgentTaskPool-2:null) =
Agent status update: [id =3D 1; name =3D apache-81-3; old status =3D Connec=
ting; event =3D AgentConnected; new status =3D Connecting; old update count=
 =3D 3; new update count =3D 4]
> 2013-08-26 06:46:54,946 DEBUG [cloud.host.Status] (AgentTaskPool-1:null) =
Agent status update: [id =3D 1; name =3D apache-81-3; old status =3D Connec=
ting; event =3D Ready; new status =3D Up; old update count =3D 4; new updat=
e count =3D 5]
> There appear to be two problems:
> 1. addHost API returns successfully even if the host has transitioned to =
'Alert' state
> 2. Something wrong with Xenserver 6.2/6.1 hosts that makes this happen
> This is on the latest off the 4.2 branch (8689f7f).
> Attaching management server log and Database Dump.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira