cloudstack-users-cn mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "赵涛" <zhaotao...@qq.com>
Subject 注册模板后出现状态:Storage agent or storage VM disconnected
Date Tue, 15 Jul 2014 03:46:24 GMT
注册模板后出现状态:Storage agent or storage VM disconnected,删除SSVM后问题依旧,请大神指导

删除SSVM虚拟机后重建的SSVM虚拟机上报日志:


2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null)
create ClusteredAgentAttache for 41
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: SecurityGroupListener
2014-07-15 10:41:46,685 INFO  [network.security.SecurityGroupListener] (AgentManager-Handler-7:null)
Received a host startup notification
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: StoragePoolMonitor
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: SecondaryStorageListener
2014-07-15 10:41:46,687 INFO  [storage.secondary.SecondaryStorageListener] (AgentManager-Handler-7:null)
Received a host startup notification com.cloud.agent.api.StartupSecondaryStorageCommand
2014-07-15 10:41:46,714 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640129: Sending  { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}]
}
2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null)
Ping from 41
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq
42-1808072705: Processing:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
[{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072705: Received:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, {
SecStorageSetupAnswer } }
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-10:null)
Seq 42-1808072705: No more commands found
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:46,809 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null)
Successfully programmed secondary storage nfs://10.11.18.7/export/d3secondary in secondary
storage VM s-909-VM
2014-07-15 10:41:46,831 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
41-766640130: Sending  { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
}
2014-07-15 10:41:46,880 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq
41-766640130: Processing:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}]
}
2014-07-15 10:41:46,881 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
41-766640130: Received:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer
} }
2014-07-15 10:41:46,881 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null)
Successfully programmed firewall rules into SSVM s-889-VM
2014-07-15 10:41:46,891 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072706: Sending  { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
}
2014-07-15 10:41:46,901 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq
42-1808072706: Processing:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10,
[{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,902 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072706: Received:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, {
Answer } }
2014-07-15 10:41:46,902 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null)
Successfully programmed firewall rules into s-909-VM
2014-07-15 10:41:46,929 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072707: Sending  { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
}
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq
42-1808072707: Processing:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
[{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-11:null)
Seq 42-1808072707: No more commands found
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072707: Received:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, {
Answer } }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand: 
2014-07-15 10:41:47,083 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null)
Successfully programmed http auth into s-909-VM
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Connect to listener: DownloadListener
2014-07-15 10:41:47,101 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072708: Sending  { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}]
}
2014-07-15 10:41:47,629 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq
42-1808072708: Processing:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
[{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-3:null) Seq
42-1808072708: No more commands found
2014-07-15 10:41:47,630 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072708: Received:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, {
ListTemplateAnswer } }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:47,653 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync found routing-3 already in the template host table
2014-07-15 10:41:47,705 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:47,771 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync did not find centos56-x86_64-xen ready on server 11, will request download to
start/resume shortly
2014-07-15 10:41:47,775 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync did not find centos53-x64 ready on server 11, will request download to start/resume
shortly
2014-07-15 10:41:47,785 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync found 217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host
table
2014-07-15 10:41:47,837 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync found 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host
table
2014-07-15 10:41:47,885 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host
table
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-6:null) Seq
41-766640129: Processing:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
[{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-6:null) Seq
41-766640129: No more commands found
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640129: Received:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, {
SecStorageSetupAnswer } }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:47,933 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null)
Template Sync found 275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host
table
2014-07-15 10:41:47,946 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null)
Successfully programmed secondary storage nfs://10.11.18.7/export/d2secondary in secondary
storage VM s-889-VM
2014-07-15 10:41:47,968 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
42-1808072709: Sending  { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
}
2014-07-15 10:41:47,976 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq
42-1808072709: Processing:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10,
[{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:47,977 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
42-1808072709: Received:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, {
Answer } }
2014-07-15 10:41:47,977 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null)
Successfully programmed firewall rules into SSVM s-909-VM
2014-07-15 10:41:47,985 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640131: Sending  { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
}
2014-07-15 10:41:48,016 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072710: Sending  { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}]
}
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq
41-766640131: Processing:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}]
}
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640131: Received:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer
} }
2014-07-15 10:41:48,033 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null)
Successfully programmed firewall rules into s-889-VM
2014-07-15 10:41:48,059 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640132: Sending  { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
}
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq
42-1808072710: Processing:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
[{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-12:null)
Seq 42-1808072710: No more commands found
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
42-1808072710: Received:  { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, {
ListVolumeAnswer } }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Monitor DownloadListener says there is an error in the connect process for 42 due to null
java.lang.NullPointerException
    at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
    at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
    at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
    at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
    at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
    at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
    at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
    at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
    at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
    at com.cloud.utils.nio.Task.run(Task.java:83)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,089 INFO  [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Host 42 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
The next status of agent 42is Alert, current status is Connecting
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Deregistering link for 42 with state Alert
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Remove Agent : 42
2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-2:null)
Processing Disconnect.
2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-2:null)
Disconnected called on 42 with status Alert
2014-07-15 10:41:48,101 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Transition:[Resource
state = Enabled, Agent event = AgentDisconnected, Host id = 42, name = s-909-VM]
2014-07-15 10:41:48,163 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Agent status
update: [id = 42; name = s-909-VM; old status = Connecting; event = AgentDisconnected; new
status = Alert; old update count = 209; new update count = 210]
2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null)
Notifying other nodes of to disconnect
2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Can not send command com.cloud.agent.api.ReadyCommand due to Host 42 is not up
2014-07-15 10:41:48,178 WARN  [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
Unable to create attache for agent: Seq 42-208:  { Cmd , MgmtId: -1, via: 42, Ver: v1, Flags:
101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}]
}
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq
41-766640132: Processing:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
[{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-9:null) Seq
41-766640132: No more commands found
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640132: Received:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, {
Answer } }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand: 
2014-07-15 10:41:48,208 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null)
Successfully programmed http auth into s-889-VM
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Connect to listener: DownloadListener
2014-07-15 10:41:48,227 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640133: Sending  { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}]
}
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq
41-766640133: Processing:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
[{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-1:null) Seq
41-766640133: No more commands found
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640133: Received:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, {
ListTemplateAnswer } }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:48,521 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found routing-3 already in the template host table
2014-07-15 10:41:48,567 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:48,609 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync did not find centos56-x86_64-xen ready on server 2, will request download to
start/resume shortly
2014-07-15 10:41:48,613 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync did not find centos53-x64 ready on server 2, will request download to start/resume
shortly
2014-07-15 10:41:48,625 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found 205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host
table
2014-07-15 10:41:48,687 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found 206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host
table
2014-07-15 10:41:48,747 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found 5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
2014-07-15 10:41:48,813 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
2014-07-15 10:41:48,860 INFO  [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null)
Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host
table
2014-07-15 10:41:48,922 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640134: Sending  { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}]
}
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq
41-766640134: Processing:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
[{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null)
Seq 41-766640134: No more commands found
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq
41-766640134: Received:  { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, {
ListVolumeAnswer } }
2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Monitor DownloadListener says there is an error in the connect process for 41 due to null
java.lang.NullPointerException
    at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
    at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
    at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
    at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
    at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
    at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
    at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
    at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
    at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
    at com.cloud.utils.nio.Task.run(Task.java:83)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,951 INFO  [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Host 41 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
The next status of agent 41is Alert, current status is Connecting
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Deregistering link for 41 with state Alert
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Remove Agent : 41
2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-7:null)
Processing Disconnect.
2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-7:null)
Disconnected called on 41 with status Alert
2014-07-15 10:41:48,964 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Transition:[Resource
state = Enabled, Agent event = AgentDisconnected, Host id = 41, name = s-889-VM]
2014-07-15 10:41:49,006 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Agent status
update: [id = 41; name = s-889-VM; old status = Connecting; event = AgentDisconnected; new
status = Alert; old update count = 391; new update count = 392]
2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null)
Notifying other nodes of to disconnect
2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Can not send command com.cloud.agent.api.ReadyCommand due to Host 41 is not up
2014-07-15 10:41:49,015 WARN  [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null)
Unable to create attache for agent: Seq 41-390:  { Cmd , MgmtId: -1, via: 41, Ver: v1, Flags:
101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}]
}
2014-07-15 10:41:50,555 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Zone 1 is ready to launch secondary storage VM
2014-07-15 10:41:50,609 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Zone 2 is ready to launch secondary storage VM
2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null)
Zone 1 is ready to launch console proxy
2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null)
Zone 2 is ready to launch console proxy
2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector
is running...
2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There
is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d2secondary
2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There
is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d3secondary
2014-07-15 10:41:51,928 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null)
Found 0 routers. 
2014-07-15 10:41:51,991 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215195:
Received:  { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2014-07-15 10:41:52,059 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 9-1410599594:
Received:  { Ans: , MgmtId: 73548179740843, via: 9, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2014-07-15 10:41:52,128 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747056:
Received:  { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2014-07-15 10:41:52,150 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 16-940116823:
Received:  { Ans: , MgmtId: 73548179740843, via: 16, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2014-07-15 10:41:52,231 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747057:
Received:  { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2014-07-15 10:41:52,312 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215196:
Received:  { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null)
Ping from 23
2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null)
Ping from 8
2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null)
Ping from 40‍
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message