cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam <adam.scarce...@gmail.com>
Subject Upgrade from 4.1.1 to 4.2 failed. Could not restart system VMs. Roll back to 4.1.1 backup initially failed: could not start management service. Finally got it to work. I hope this helps someone else avoid my same pain...
Date Tue, 15 Oct 2013 21:10:38 GMT
I upgraded my 4 host private dev cloud running on CentOS 6.4 x86_64 from
version 4.1.1 to 4.2 per upgrade instructions. I'm just using 4 HP Z 600
workstations (box 1 has two nics, the cloudstack-management server, the
mysql db, primary and secondary storage and a cloudstack-agent, and boxes 2
- 4 each just have primary storage and a cloudstack-agent). It's nothing
fancy, but it's been working perfectly now for months. All seemed to go
very smoothly except for the very last step:

{code}
nohup cloudstack-sysvmadm -d cs-east-dev1 -u root -p support -a > sysvm.log
2>&1 &
{code}


It could not restart the system vms for some reason. (I do not have the
original sysvm.log as I've now been playing with this failed upgrade for
two days). However, here is the sysvm.log from the very last attempt:

{code}
nohup: ignoring input

Stopping and starting 1 secondary storage vm(s)...
curl: (7) couldn't connect to host
ERROR: Failed to stop secondary storage vm with id 14

Done stopping and starting secondary storage vm(s)

Stopping and starting 0 console proxy vm(s)...
No running console proxy vms found


Stopping and starting 1 running routing vm(s)...
curl: (7) couldn't connect to host
2
Done restarting router(s).
{code}


As I mentioned above, I've been playing around with this for 2 days now and
actually got the 4.2 management server to finally start, but none of the
System VMs worked. I even restarted all of the CentOS hosts (which was a
huge hassle), but that didn't seem to help at all. I eventually found this
bug: https://issues.apache.org/jira/browse/CLOUDSTACK-4826 which seemed
similar to my issues.

I was also experiencing a strange issue where all 10 of my private
Management IP Addresses were used for some reason. Every time I restarted
the cloudstack-management service, 2 more IPs were taken up, but none ever
got released. Also since the System VMs would not start, my secondary
storage wouldn't start up either.

About an hour ago I gave up on 4.2 and I decided to roll back to 4.1.1 on
all 4 workstations. First I shutdown all VM instances, then stopped all
cloudstack-* services on all 4 workstations, and then ran a "yum downgrade
cloudstack-*" on all 4 workstations:

{code}
[root@cs-east-dev1 yum.repos.d]# yum downgrade cloudstack-*
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Downgrade Process
Loading mirror speeds from cached hostfile
 * base: centos.mirror.nac.net
 * extras: mirror.trouble-free.net
 * rpmforge: mirror.us.leaseweb.net
 * updates: mirror.cogentco.com
Resolving Dependencies
--> Running transaction check
---> Package cloudstack-agent.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-agent.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-awsapi.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-awsapi.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-cli.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-cli.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-common.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-common.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-management.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-management.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-usage.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-usage.x86_64 0:4.2.0-1.el6 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================
 Package                                       Arch
  Version                               Repository
 Size
=============================================================================================================================================================
Downgrading:
 cloudstack-agent                              x86_64
  4.1.1-0.el6                           cloudstack
 37 M
 cloudstack-awsapi                             x86_64
  4.1.1-0.el6                           cloudstack
 56 M
 cloudstack-cli                                x86_64
  4.1.1-0.el6                           cloudstack
 32 k
 cloudstack-common                             x86_64
  4.1.1-0.el6                           cloudstack
 92 M
 cloudstack-management                         x86_64
  4.1.1-0.el6                           cloudstack
 55 M
 cloudstack-usage                              x86_64
  4.1.1-0.el6                           cloudstack
 37 M

Transaction Summary
=============================================================================================================================================================
Downgrade     6 Package(s)

Total download size: 278 M
Is this ok [y/N]: y
Downloading Packages:
(1/6): cloudstack-agent-4.1.1-0.el6.x86_64.rpm
                                                           |  37 MB
00:13
(2/6): cloudstack-awsapi-4.1.1-0.el6.x86_64.rpm
                                                          |  56 MB     01:01
(3/6): cloudstack-cli-4.1.1-0.el6.x86_64.rpm
                                                           |  32 kB
00:00
(4/6): cloudstack-common-4.1.1-0.el6.x86_64.rpm
                                                          |  92 MB     01:53
(5/6): cloudstack-management-4.1.1-0.el6.x86_64.rpm
                                                          |  55 MB     00:39
(6/6): cloudstack-usage-4.1.1-0.el6.x86_64.rpm
                                                           |  37 MB
00:12
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total
                                                 1.2 MB/s | 278 MB     04:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : cloudstack-common-4.1.1-0.el6.x86_64

1/12
  Installing : cloudstack-management-4.1.1-0.el6.x86_64

2/12
warning: /etc/cloudstack/management/db.properties created as
/etc/cloudstack/management/db.properties.rpmnew
ln: creating symbolic link
`/usr/share/cloudstack-management/webapps7080/webapps': File exists
Please download vhd-util from
http://download.cloud.com.s3.amazonaws.com/tools/vhd-util and put it in
/usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/
Unable to determine ssl settings for server.xml, please run
cloudstack-setup-management manually
Unable to determine ssl settings for tomcat.conf, please run
cloudstack-setup-management manually
  Installing : cloudstack-awsapi-4.1.1-0.el6.x86_64

3/12
  Installing : cloudstack-agent-4.1.1-0.el6.x86_64

 4/12
warning: /etc/cloudstack/agent/agent.properties created as
/etc/cloudstack/agent/agent.properties.rpmnew
  Installing : cloudstack-cli-4.1.1-0.el6.x86_64

 5/12
  Installing : cloudstack-usage-4.1.1-0.el6.x86_64

 6/12
Replacing db.properties with management server db.properties
  Cleanup    : cloudstack-awsapi-4.2.0-1.el6.x86_64

7/12
  Cleanup    : cloudstack-management-4.2.0-1.el6.x86_64

8/12
Stopping cloudstack-management:[  OK  ]
  Cleanup    : cloudstack-agent-4.2.0-1.el6.x86_64

 9/12
Stopping Cloud Agent:
  Cleanup    : cloudstack-common-4.2.0-1.el6.x86_64

 10/12
  Cleanup    : cloudstack-cli-4.2.0-1.el6.x86_64

11/12
  Cleanup    : cloudstack-usage-4.2.0-1.el6.x86_64

12/12
  Verifying  : cloudstack-usage-4.1.1-0.el6.x86_64

 1/12
  Verifying  : cloudstack-awsapi-4.1.1-0.el6.x86_64

2/12
  Verifying  : cloudstack-management-4.1.1-0.el6.x86_64

3/12
  Verifying  : cloudstack-cli-4.1.1-0.el6.x86_64

 4/12
  Verifying  : cloudstack-common-4.1.1-0.el6.x86_64

5/12
  Verifying  : cloudstack-agent-4.1.1-0.el6.x86_64

 6/12
  Verifying  : cloudstack-management-4.2.0-1.el6.x86_64

7/12
  Verifying  : cloudstack-cli-4.2.0-1.el6.x86_64

 8/12
  Verifying  : cloudstack-common-4.2.0-1.el6.x86_64

9/12
  Verifying  : cloudstack-awsapi-4.2.0-1.el6.x86_64

 10/12
  Verifying  : cloudstack-usage-4.2.0-1.el6.x86_64

11/12
  Verifying  : cloudstack-agent-4.2.0-1.el6.x86_64

12/12

Removed:
  cloudstack-agent.x86_64 0:4.2.0-1.el6
 cloudstack-awsapi.x86_64 0:4.2.0-1.el6
cloudstack-cli.x86_64 0:4.2.0-1.el6
  cloudstack-common.x86_64 0:4.2.0-1.el6
cloudstack-management.x86_64 0:4.2.0-1.el6
cloudstack-usage.x86_64 0:4.2.0-1.el6

Installed:
  cloudstack-agent.x86_64 0:4.1.1-0.el6
 cloudstack-awsapi.x86_64 0:4.1.1-0.el6
cloudstack-cli.x86_64 0:4.1.1-0.el6
  cloudstack-common.x86_64 0:4.1.1-0.el6
cloudstack-management.x86_64 0:4.1.1-0.el6
cloudstack-usage.x86_64 0:4.1.1-0.el6

Complete!
[root@cs-east-dev1 yum.repos.d]# service cloudstack-agent start
Starting Cloud Agent:
[root@cs-east-dev1 yum.repos.d]# service cloudstack-management start
Starting cloudstack-management:                            [  OK  ]
[root@cs-east-dev1 yum.repos.d]
{code}


*NOTE*: On the other 3 hosts it only needed to downgrade cloudstack-agent
and cloudstack-common.


After the downgrade I received this silly error in the management-server.log

{code}
[root@cs-east-dev1 yum.repos.d]# tail -f
/var/log/cloudstack/management/management-server.log
2013-10-15 15:36:20,138 INFO  [web.context.ContextLoader] (main:null) Root
WebApplicationContext: initialization started
2013-10-15 15:36:20,220 INFO  [context.support.XmlWebApplicationContext]
(main:null) Refreshing Root WebApplicationContext: startup date [Tue Oct 15
15:36:20 EDT 2013]; root of context hierarchy
2013-10-15 15:36:20,276 INFO  [factory.xml.XmlBeanDefinitionReader]
(main:null) Loading XML bean definitions from class path resource
[applicationContext.xml]
2013-10-15 15:36:20,713 INFO  [factory.xml.XmlBeanDefinitionReader]
(main:null) Loading XML bean definitions from class path resource
[nonossComponentContext.xml]
2013-10-15 15:36:22,130 INFO
 [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:22,162 INFO  [context.support.XmlWebApplicationContext]
(main:null) Bean 'transactionContextBuilder' of type [class
com.cloud.utils.db.TransactionContextBuilder] is not eligible for getting
processed by all BeanPostProcessors (for example: not eligible for
auto-proxying)
2013-10-15 15:36:22,167 INFO  [context.support.XmlWebApplicationContext]
(main:null) Bean 'actionEventInterceptor' of type [class
com.cloud.event.ActionEventInterceptor] is not eligible for getting
processed by all BeanPostProcessors (for example: not eligible for
auto-proxying)
2013-10-15 15:36:22,212 INFO  [factory.support.DefaultListableBeanFactory]
(main:null) Pre-instantiating singletons in
org.springframework.beans.factory.support.DefaultListableBeanFactory@757df1b6:
defining beans
[org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,componentContext,transactionContextBuilder,actionEventInterceptor,instantiatePostProcessor,onwireRegistry,messageSerializer,transportProvider,rpcProvider,eventBus,serviceOfferingDaoImpl,diskOfferingDaoImpl,dataCenterDaoImpl,hostPodDaoImpl,vlanDaoImpl,userDaoImpl,VMTemplateDaoImpl,hypervisorCapabilitiesDaoImpl,accountDaoImpl,accountDetailsDaoImpl,accountJoinDaoImpl,accountVlanMapDaoImpl,agentUpgradeDaoImpl,alertDaoImpl,asyncJobDaoImpl,asyncJobJoinDaoImpl,autoScalePolicyConditionMapDaoImpl,autoScalePolicyDaoImpl,autoScaleVmGroupDaoImpl,autoScaleVmGroupPolicyMapDaoImpl,autoScaleVmProfileDaoImpl,capacityDaoImpl,certificateDaoImpl,clusterDaoImpl,clusterDetailsDaoImpl,commandExecLogDaoImpl,conditionDaoImpl,consoleProxyDaoImpl,counterDaoImpl,dataCenterIpAddressDaoImpl,dataCenterJoinDaoImpl,dataCenterLinkLocalIpAddressDaoImpl,dataCenterVnetDaoImpl,dataStoreProviderDaoImpl,dcDetailsDaoImpl,engineDcDetailsDaoImpl,diskOfferingJoinDaoImpl,domainDaoImpl,domainRouterDaoImpl,domainRouterJoinDaoImpl,elasticLbVmMapDaoImpl,engineClusterDaoImpl,engineDataCenterDaoImpl,engineHostDaoImpl,engineHostPodDaoImpl,eventDaoImpl,eventJoinDaoImpl,externalFirewallDeviceDaoImpl,externalLoadBalancerDeviceDaoImpl,externalPublicIpStatisticsDaoImpl,firewallRulesCidrsDaoImpl,firewallRulesDaoImpl,guestOSCategoryDaoImpl,guestOSDaoImpl,highAvailabilityDaoImpl,hostDaoImpl,hostDetailsDaoImpl,engineHostDetailsDaoImpl,hostJoinDaoImpl,engineHostTagsDaoImpl,hostTagsDaoImpl,hostTransferMapDaoImpl,iPAddressDaoImpl,identityDaoImpl,imageDaoStoreDaoImpl,imageDataDaoImpl,imageDataStoreProviderDaoImpl,inlineLoadBalancerNicMapDaoImpl,instanceGroupDaoImpl,instanceGroupJoinDaoImpl,instanceGroupVMMapDaoImpl,itWorkDaoImpl,keystoreDaoImpl,lBStickinessPolicyDaoImpl,launchPermissionDaoImpl,loadBalancerDaoImpl,loadBalancerVMMapDaoImpl,managementServerHostDaoImpl,managementServerHostPeerDaoImpl,networkAccountDaoImpl,networkDaoImpl,networkDomainDaoImpl,networkExternalFirewallDaoImpl,networkExternalLoadBalancerDaoImpl,networkOfferingDaoImpl,networkOfferingServiceMapDaoImpl,networkOpDaoImpl,networkRuleConfigDaoImpl,networkServiceMapDaoImpl,nicDaoImpl,objectInDataStoreDaoImpl,ovsTunnelInterfaceDaoImpl,ovsTunnelNetworkDaoImpl,physicalNetworkDaoImpl,physicalNetworkIsolationMethodDaoImpl,physicalNetworkServiceProviderDaoImpl,physicalNetworkTagDaoImpl,physicalNetworkTrafficTypeDaoImpl,podVlanDaoImpl,podVlanMapDaoImpl,portForwardingRulesDaoImpl,primaryDataStoreDaoImpl,primaryDataStoreDetailsDaoImpl,privateIpDaoImpl,projectAccountDaoImpl,projectAccountJoinDaoImpl,projectDaoImpl,projectInvitationDaoImpl,projectInvitationJoinDaoImpl,projectJoinDaoImpl,regionDaoImpl,remoteAccessVpnDaoImpl,resourceCountDaoImpl,resourceLimitDaoImpl,resourceTagJoinDaoImpl,resourceTagsDaoImpl,routerNetworkDaoImpl,s3DaoImpl,sSHKeyPairDaoImpl,secondaryStorageVmDaoImpl,securityGroupDaoImpl,securityGroupJoinDaoImpl,securityGroupRuleDaoImpl,securityGroupRulesDaoImpl,securityGroupVMMapDaoImpl,securityGroupWorkDaoImpl,serviceOfferingJoinDaoImpl,site2SiteCustomerGatewayDaoImpl,site2SiteVpnConnectionDaoImpl,site2SiteVpnGatewayDaoImpl,snapshotDao2Impl,snapshotDaoImpl,snapshotPolicyDaoImpl,snapshotScheduleDaoImpl,stackMaidDaoImpl,staticRouteDaoImpl,storageNetworkIpAddressDaoImpl,storageNetworkIpRangeDaoImpl,storagePoolDaoImpl,storagePoolDetailsDaoImpl,storagePoolHostDaoImpl,storagePoolJoinDaoImpl,storagePoolWorkDaoImpl,swiftDaoImpl,syncQueueDaoImpl,syncQueueItemDaoImpl,templatePrimaryDataStoreDaoImpl,uploadDaoImpl,usageDaoImpl,usageEventDaoImpl,usageIPAddressDaoImpl,usageJobDaoImpl,usageLoadBalancerPolicyDaoImpl,usageNetworkDaoImpl,usageNetworkOfferingDaoImpl,usagePortForwardingRuleDaoImpl,usageSecurityGroupDaoImpl,usageStorageDaoImpl,usageVMInstanceDaoImpl,usageVPNUserDaoImpl,usageVolumeDaoImpl,userAccountDaoImpl,userAccountJoinDaoImpl,userIpv6AddressDaoImpl,userStatisticsDaoImpl,userStatsLogDaoImpl,userVmDetailsDaoImpl,userVmJoinDaoImpl,vMComputeTagDaoImpl,vMEntityDaoImpl,vMInstanceDaoImpl,vMNetworkMapDaoImpl,vMReservationDaoImpl,vMRootDiskTagDaoImpl,vMTemplateDetailsDaoImpl,vMTemplateHostDaoImpl,vMTemplatePoolDaoImpl,vMTemplateS3DaoImpl,vMTemplateSwiftDaoImpl,vMTemplateZoneDaoImpl,versionDaoImpl,virtualRouterProviderDaoImpl,vmRulesetLogDaoImpl,volumeDao2Impl,volumeDaoImpl,volumeHostDaoImpl,volumeJoinDaoImpl,volumeReservationDaoImpl,vpcDaoImpl,vpcGatewayDaoImpl,vpcOfferingDaoImpl,vpcOfferingServiceMapDaoImpl,vpnUserDaoImpl,userVmDaoImpl,portProfileDaoImpl,clusterVSMMapDaoImpl,encryptionSecretKeyChecker,StaticRoleBasedAPIAccessChecker,databaseIntegrityChecker,domainChecker,basicAgentAuthManager,MD5UserAuthenticator,LDAPUserAuthenticator,PlainTextUserAuthenticator,Ovs,SecurityGroupProvider,VirtualRouter,VpcVirtualRouter,elasticLoadBalancerElement,firstFitAllocator,randomAllocator,FirstFitRouting,LocalStoragePoolAllocator,FirstFitStoragePoolAllocator,UserConcentratedAllocator,useLocalForRootAllocator,garbageCollectingStoragePoolAllocator,randomStoragePoolAllocator,recreatableFencer,recreateHostAllocator,secondaryStorageVmDefaultAllocator,ConsoleProxyAllocator,ExteralIpAddressAllocator,hyervisorTemplateAdapter,clusterAlertAdapter,consoleProxyAlertAdapter,secondaryStorageVmAlertAdapter,clusterServiceServletAdapter,CheckOnAgentInvestigator,XenServerInvestigator,UserVmDomRInvestigator,ManagementIPSystemVMInvestigator,XenServerFencer,KVMFencer,OvmFencer,XcpServerDiscoverer,SecondaryStorageDiscoverer,KvmServerDiscoverer,HypervServerDiscoverer,OvmDiscoverer,dummyHostDiscoverer,UserDispersingPlanner,UserConcentratedPodPlanner,clusterBasedAgentLoadBalancerPlanner,StorageNetworkGuru,ExternalGuestNetworkGuru,PublicNetworkGuru,PodBasedNetworkGuru,ControlNetworkGuru,DirectNetworkGuru,DirectPodBasedNetworkGuru,OvsGuestNetworkGuru,PrivateNetworkGuru,XenServerGuru,KVMGuru,OvmGuru,HypervisorPlannerSelector,accountManagerImpl,managementServerImpl,storageManagerImpl,FirstFitPlanner,resourceManagerImpl,highAvailabilityManagerExtImpl,clusteredVirtualMachineManagerImpl,networkManagerImpl,networkModelImpl,configurationServerImpl,clusterManagerImpl,clusteredAgentManagerImpl,userVmManagerImpl,consoleProxyManagerImpl,securityGroupManagerImpl2,premiumSecondaryStorageManagerImpl,ipv6AddressManagerImpl,apiRateLimitServiceImpl,alertManagerImpl,asyncJobExecutorContextImpl,asyncJobManagerImpl,autoScaleManagerImpl,capacityManagerImpl,clusterFenceManagerImpl,configurationManagerImpl,dataStoreProviderManagerImpl,domainManagerImpl,downloadMonitorImpl,elasticLoadBalancerManagerImpl,entityManagerImpl,externalLoadBalancerUsageManagerImpl,externalNetworkDeviceManagerImpl,firewallManagerImpl,hypervisorGuruManagerImpl,identityServiceImpl,keystoreManagerImpl,loadBalancingRulesManagerImpl,networkACLManagerImpl,networkServiceImpl,networkUsageManagerImpl,oCFS2ManagerImpl,ovsTunnelManagerImpl,projectManagerImpl,queryManagerImpl,regionManagerImpl,regionServiceImpl,remoteAccessVpnManagerImpl,resourceLimitManagerImpl,rulesManagerImpl,s3ManagerImpl,site2SiteVpnManagerImpl,snapshotManagerImpl,snapshotSchedulerImpl,storageNetworkManagerImpl,swiftManagerImpl,syncQueueManagerImpl,taggedResourceManagerImpl,templateManagerImpl,upgradeManagerImpl,uploadMonitorImpl,usageServiceImpl,virtualNetworkApplianceManagerImpl,vpcManagerImpl,vpcVirtualNetworkApplianceManagerImpl,cloudZonesStartupProcessor,managementServerNode,actionEventUtils,agentMonitor,alertGenerator,apiDBUtils,apiDiscoveryServiceImpl,apiDispatcher,apiResponseHelper,apiServer,apiServlet,cloudOrchestrator,clusterRestService,consoleProxyServlet,dataCenterResourceManagerImpl,dataDisk,dataMotionServiceImpl,dataObjectManagerImpl,dataStoreManagerImpl,defaultEndPointSelector,defaultImageDataStoreProvider,defaultImageMotionStrategy,defaultPrimaryDataStoreProviderManagerImpl,defaultPrimaryDatastoreProviderImpl,hypervisorBasedSnapshot,hypervsiorHostEndPointRpcServer,iSCSI,ISO,imageDataFactoryImpl,imageDataManagerImpl,imageDataStoreHelper,imageDataStoreManagerImpl,imageFormatHelper,imageMotionServiceImpl,imageServiceImpl,iso,networkFileSystem,networkRestService,OVA,objectInDataStoreManagerImpl,podRestService,primaryDataStoreHelper,provisioningServiceImpl,QCOW2,registerCompleteServlet,rootDisk,snapshotDataFactoryImpl,snapshotServiceImpl,statsCollector,templateInstallStrategyImpl,unknown,usageEventUtils,userContextInitializer,VHD,vMEntityManagerImpl,virtualMachineEntityImpl,virtualMachineRestService,volumeDataFactoryImpl,volumeManagerImpl,volumeRestService,volumeServiceImpl,volumeTypeHelper,zoneRestService,premiumDatabaseUpgradeChecker,configurationDaoImpl,VmwareFencer,vmwareServerDiscoverer,VMwareGuru,VmwareInvestigator,VmwareManager,vmwareContextFactory,niciraNvpDaoImpl,niciraNvpNicMappingDaoImpl,niciraNvpRouterMappingDaoImpl,NiciraNvpGuestNetworkGuru,NiciraNvp,lunDaoImpl,poolDaoImpl,netappVolumeDaoImpl,NetappManager,JuniperSRX,netScalerPodDaoImpl,Netscaler,F5BigIP,ciscoNexusVSMDeviceDaoImpl,CiscoNexus1000vVSM,bigSwitchVnsDaoImpl,bigSwitchVnsElement,userAuthenticators,securityCheckers,resourceDiscoverers,haInvestigators,haFenceBuilders,deploymentPlanners,podAllocators,hostAllocators,storagePoolAllocators,ipDeployers,dhcpProviders,networkGurus,networkElements,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0];
root of factory hierarchy
2013-10-15 15:36:22,219 INFO  [utils.component.ComponentContext]
(main:null) Setup Spring Application context
2013-10-15 15:36:22,515 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
(main:null) Encryption Type: file
2013-10-15 15:36:27,126 INFO  [cloud.serializer.GsonHelper] (main:null)
Default Builder inited.
2013-10-15 15:36:27,901 INFO  [hypervisor.vmware.VmwareServerDiscoverer]
(main:null) VmwareServerDiscoverer is constructed
2013-10-15 15:36:28,117 INFO  [web.context.ContextLoader] (main:null) Root
WebApplicationContext: initialization completed in 7978 ms
2013-10-15 15:36:28,134 INFO  [cloud.utils.LogUtils] (main:null) log4j
configuration found at /etc/cloudstack/management/log4j-cloud.xml
2013-10-15 15:36:28,151 INFO
 [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:28,152 INFO
 [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:28,153 INFO
 [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:33,514 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackAccountDaoImpl_EnhancerByCloudStack_3de75b6e
2013-10-15 15:36:33,516 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.OfferingDaoImpl_EnhancerByCloudStack_d2f1cac8
2013-10-15 15:36:33,517 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SMetaDaoImpl_EnhancerByCloudStack_ef98bac0
2013-10-15 15:36:33,517 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.UserCredentialsDaoImpl_EnhancerByCloudStack_909a85df
2013-10-15 15:36:33,517 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MultipartMetaDaoImpl_EnhancerByCloudStack_d7013d7f
2013-10-15 15:36:33,517 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackConfigurationDaoImpl_EnhancerByCloudStack_fecccd9d
2013-10-15 15:36:33,517 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.BucketPolicyDaoImpl_EnhancerByCloudStack_56673762
2013-10-15 15:36:33,517 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SObjectItemDaoImpl_EnhancerByCloudStack_3a79ef2b
2013-10-15 15:36:33,518 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MHostMountDaoImpl_EnhancerByCloudStack_8f7d233a
2013-10-15 15:36:33,518 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MHostDaoImpl_EnhancerByCloudStack_165457b
2013-10-15 15:36:33,518 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MultiPartUploadsDaoImpl_EnhancerByCloudStack_bcd9eaa8
2013-10-15 15:36:33,518 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SBucketDaoImpl_EnhancerByCloudStack_8e162773
2013-10-15 15:36:33,518 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SAclDaoImpl_EnhancerByCloudStack_104ffef
2013-10-15 15:36:33,519 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MultiPartPartsDaoImpl_EnhancerByCloudStack_88d4152a
2013-10-15 15:36:33,519 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackSvcOfferingDaoImpl_EnhancerByCloudStack_882a018d
2013-10-15 15:36:33,519 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_29f7bbc5
2013-10-15 15:36:33,519 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStack_c20b110
2013-10-15 15:36:33,519 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_10d22bb6
2013-10-15 15:36:33,520 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_635f1066
2013-10-15 15:36:33,561 INFO  [utils.component.ComponentContext]
(Timer-1:null) Running SystemIntegrityChecker encryptionSecretKeyChecker
2013-10-15 15:36:33,563 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
(Timer-1:null) Encryption Type: file
2013-10-15 15:36:33,563 WARN  [utils.crypt.EncryptionSecretKeyChecker]
(Timer-1:null) Encryption already enabled, is check() called twice?
2013-10-15 15:36:33,564 INFO  [utils.component.ComponentContext]
(Timer-1:null) Running SystemIntegrityChecker databaseIntegrityChecker
2013-10-15 15:36:33,564 INFO  [cloud.upgrade.DatabaseIntegrityChecker]
(Timer-1:null) Grabbing lock to check for database integrity.
2013-10-15 15:36:33,750 INFO  [cloud.upgrade.DatabaseIntegrityChecker]
(Timer-1:null) Performing database integrity check
2013-10-15 15:36:33,754 INFO  [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloudStack_8e8faadb
2013-10-15 15:36:33,763 DEBUG [cloud.upgrade.DatabaseIntegrityChecker]
(Timer-1:null) No duplicate hosts with the same local storage found in
database
2013-10-15 15:36:33,767 DEBUG [upgrade.dao.VersionDaoImpl] (Timer-1:null)
Checking to see if the database is at a version before it was the version
table is created
2013-10-15 15:36:33,787 INFO  [utils.component.ComponentContext]
(Timer-1:null) Running SystemIntegrityChecker managementServerNode
2013-10-15 15:36:33,788 ERROR [utils.component.ComponentContext]
(Timer-1:null) *System integrity check failed. Refuse to startup*
[root@cs-east-dev1 yum.repos.d]#
{code}


I googled this issue and found the following discussion:

http://mail-archives.apache.org/mod_mbox/cloudstack-issues/201309.mbox/%3CJIRA.12671428.1380572215050.5987.1380572367958@arcas%3E


and this discussion points to the following defects:

https://issues.apache.org/jira/browse/CLOUDSTACK-4770 &&
https://issues.apache.org/jira/browse/CLOUDSTACK-4170


But defect CLOUDSTACK-4770 is set to *minor *and there doesn't seem to be
any activity on it.

In the CLOUDSTACK-4770 defect report Richard Chatterton mentions having a
MAC address of 00:00:00:00:00:00, so I performed the same steps as Richard
(except for version 4.1.1) and got the following output:

{code}
[root@cs-east-dev1 ~]# java -classpath
/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/cloud-utils-4.1.1.jar
com.cloud.utils.net.MacAddress
*addr in integer is 0*
*addr in bytes is  0 0 0 0 0 0*
*addr in char is 00:00:00:00:00:00*
[root@cs-east-dev1 ~]# ifconfig -a
*cloud0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00*
          inet addr:169.254.0.1  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::9c89:feff:feca:45e5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1752 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:73800 (72.0 KiB)

cloudbr0  Link encap:Ethernet  HWaddr 18:A9:05:B7:44:7A
          inet addr:10.97.36.115  Bcast:10.97.36.255  Mask:255.255.255.0
          inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:36186 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3830135 (3.6 MiB)  TX bytes:746 (746.0 b)

eth0      Link encap:Ethernet  HWaddr 18:A9:05:B7:44:7A
          inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:44652 errors:0 dropped:0 overruns:0 frame:0
          TX packets:251 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5815431 (5.5 MiB)  TX bytes:19286 (18.8 KiB)
          Interrupt:17

eth1      Link encap:Ethernet  HWaddr 00:18:F8:0B:53:2B
          inet addr:10.97.36.105  Bcast:10.97.36.255  Mask:255.255.255.0
          inet6 addr: fe80::218:f8ff:fe0b:532b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:542135 errors:0 dropped:27 overruns:0 frame:0
          TX packets:422950 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:506474573 (483.0 MiB)  TX bytes:157977312 (150.6 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:547879 errors:0 dropped:0 overruns:0 frame:0
          TX packets:547879 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:122259530 (116.5 MiB)  TX bytes:122259530 (116.5 MiB)

virbr0    Link encap:Ethernet  HWaddr 52:54:00:40:5C:17
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

virbr0-nic Link encap:Ethernet  HWaddr 52:54:00:40:5C:17
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@cs-east-dev1 ~]#
{code}


So... then I was curious what would happen if I simply restarted the
network service:

{code}
[root@cs-east-dev1 ~]# service network status
Configured devices:
lo cloudbr0 eth0 eth1
Currently active devices:
lo eth0 eth1 cloudbr0 virbr0 cloud0
[root@cs-east-dev1 ~]# service network restart
Shutting down interface cloudbr0:                          [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]
Bringing up interface eth1:
Determining IP information for eth1... done.
                                                           [  OK  ]
Bringing up interface cloudbr0:                            [  OK  ]
[root@cs-east-dev1 ~]# ifconfig -a
cloudbr0  Link encap:Ethernet  HWaddr 18:A9:05:B7:44:7A
          inet addr:10.97.36.115  Bcast:10.97.36.255  Mask:255.255.255.0
          inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2639 (2.5 KiB)  TX bytes:566 (566.0 b)

cloud0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          inet addr:169.254.0.1  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::9c89:feff:feca:45e5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1752 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:73800 (72.0 KiB)

eth0      Link encap:Ethernet  HWaddr 18:A9:05:B7:44:7A
          inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:65 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7266 (7.0 KiB)  TX bytes:1132 (1.1 KiB)
          Interrupt:17

eth1      Link encap:Ethernet  HWaddr 00:18:F8:0B:53:2B
          inet addr:10.97.36.105  Bcast:10.97.36.255  Mask:255.255.255.0
          inet6 addr: fe80::218:f8ff:fe0b:532b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:546735 errors:0 dropped:27 overruns:0 frame:0
          TX packets:428375 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:506928715 (483.4 MiB)  TX bytes:160011479 (152.5 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:548153 errors:0 dropped:0 overruns:0 frame:0
          TX packets:548153 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:122274810 (116.6 MiB)  TX bytes:122274810 (116.6 MiB)

virbr0    Link encap:Ethernet  HWaddr 52:54:00:40:5C:17
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

virbr0-nic Link encap:Ethernet  HWaddr 52:54:00:40:5C:17
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@cs-east-dev1 ~]#
{code}



Notice how now cloudbr0 is listed first, and that has a valid MAC address.
Believe it or not, now the management server starts up fine, however... the
System VMs still wouldn't start. They were stuck in the "Starting" state.
So... I downloaded the System ISO seed again

{code}
[root@cs-east-dev1 ~]#
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
-m /export/secondary -u
http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2-h
kvm -F
--2013-10-15 16:31:45--
http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2
Resolving download.cloud.com... 207.171.187.118
Connecting to download.cloud.com|207.171.187.118|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 286036668 (273M) [binary/octet-stream]
Saving to:
â/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2â

100%[===================================================================================================================>]
286,036,668 4.15M/s   in 73s

2013-10-15 16:32:59 (3.73 MB/s) -
â/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2â
saved [286036668/286036668]

Uncompressing to
/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2.tmp
(type bz2)...could take a long time
Moving to
/export/secondary/template/tmpl/1/3///1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2...could
take a while
Successfully installed system VM template  to
/export/secondary/template/tmpl/1/3/
[root@cs-east-dev1 ~]#
{code}


Once that was complete I destroyed the system VMs and restarted
cloudstack-management. Now everything seems back to normal.

Bottom line, I don't feel that 4.2 is production ready yet for KVM on
CentOS 6.4. I hope this helps someone else...

-- 

Best Regards,



Adam Scarcella

Sr. Support Engineer | Spotfire, TIBCO Software Inc.

Web: http://spotfire.tibco.com/support

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message