incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Hartmann <mhartm...@tls.net>
Subject Re: Two Issues with XenServer 6.0.2 and CS 3.0.1
Date Wed, 16 May 2012 16:24:11 GMT
Just an FYI, I had to manually scrub the database then re-add them via 
the CS UI. Things are coming back to life very slowly. Thanks again for 
your assistance! :)

Cheers,

Matthew


On 5/16/2012 10:00 AM, Matthew Hartmann wrote:
> Well, I'm at the point I'm ready to remove Primary & Secondary 
> Storage, however CloudStack refuses to remove them even though there 
> are 0 hosts configured. CloudStack reports when trying to remove 
> Primary Storage: "Failed to delete storage pool"
>
> When trying to delete Secondary Storage CloudStack reports: "Can not 
> delete this secondary storage due to there are still snapshots on it"
>
> I disabled the Pod and Zone before attempting to remove Primary and 
> Secondary Storage, as well as putting Primary Storage in Maintenance 
> Mode.
>
> Management Server log reports when trying to delete Secondary Storage:
>
> 2012-05-16 09:57:29,553 ERROR [cloud.api.ApiDispatcher] 
> (catalina-exec-24:null) Exception while executing DeleteHostCmd:
> com.cloud.utils.exception.CloudRuntimeException: Can not delete this 
> secondary storage due to there are still snapshots on it
>
> Management Server log reports when trying to delete Primary Storage:
>
> 2012-05-16 09:58:53,600 WARN  [cloud.storage.StorageManagerImpl] 
> (catalina-exec-2:null) Cannot delete pool 
> san-p1n1c1.iedc.cloud.tls.net as there are associated vols for this pool
> 2012-05-16 09:58:53,601 WARN  [cloud.api.ApiDispatcher] 
> (catalina-exec-2:null) class com.cloud.api.ServerApiException : Failed 
> to delete storage pool
>
> I suspect manual scrubbing of the database is in order?
>
> Cheers,
>
> Matthew
>
>
> On 5/14/2012 5:17 PM, Geoff Higginbottom wrote:
>> Matthew,
>>
>> Removing and then re-adding Pri Storage will not wipe the LUNs and 
>> the existing VHD files which are there, assuming the associated Guest 
>> Instances are still in the database should also remap.
>>
>> How to avoid this in the future? It's pretty rare for a XenServer 
>> cluster to bomb out in this way, but a 'standby cluster' would not 
>> really help as that would have its own Storage Pool.  However having 
>> at least two Primary Storage pools in the Cluster (eg two iSCSI LUNs) 
>> will allow you to migrate Guest Instances between them should you 
>> detect any problems.
>>
>> Also having multiple PODs and Clusters and forcing the distribution 
>> of Guest Instances for a particular Account to be spread amongst 
>> them, rather than relying on the default 'Random' allocation policies 
>> reduces risks in the event of any failures.
>>
>> If you do have data on the existing Primary Storage and want to be 
>> über safe (my preferred stance) then copy the VHDs off first, you can 
>> always re-import them back into CloudStack via Templates if you have 
>> to, but I assume as there are no clients on this system yet the LUNs 
>> should be empty.
>>
>> Kind Regards
>>
>> Geoff Higginbottom
>> CTO / Senior Consultant
>> ShapeBlue Ltd
>>
>> geoff.higginbottom@shapeblue.com<mailto:geoff.higginbottom@shapeblue.com> 

>> | www.shapeblue.com<http://www.shapeblue.com/>
>>
>> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
>>
>>
>> On 14 May 2012, at 22:02, "Matthew 
>> Hartmann"<mhartmann@tls.net<mailto:mhartmann@tls.net>>  wrote:
>>
>> On 5/14/2012 4:48 PM, Geoff Higginbottom wrote:
>> I had assumed you had Primary and Secondary storage on different IP 
>> ranges to your Management Server which does need to be able to access 
>> Secondary Storage but not Primary Storage.
>>
>> That assumption is correct, Primary Storage is on a different subnet 
>> than Secondary Storage. Secondary Storage is sitting on the 
>> Management Network subnet.
>>
>>
>> I now note your Primary Storage is iSCSI, and that you had a failure 
>> of all Hosts.
>>
>> When adding Storage and Hosts, you have to add Hosts, before you add 
>> Primary Storage, and you have to add Primary Storage before Secondary 
>> Storage.
>>
>> This is all taken care of by the Add Zone Wizard, but if you have a 
>> failure of all Hosts in a Zone then you need to remove both Primary 
>> and Secondary Storage, then add a Host, then add Primary Storage, 
>> then add Secondary storage.
>>
>> Templates, ISOs etc that are on Secondary will disappear from the 
>> GUI, but don't worry, they will be detected by the SSVM once it comes 
>> online, and they will then reappear in the GUI.
>>
>> If this is a simple test system then it may be easier to simply drop 
>> the DB Schema, and recreate it from scratch, or create a new Zone and 
>> let the wizard take care of everything.
>>
>> This actually was a system we had *just* placed into production. We 
>> luckily did not have any clients setup on this cloud yet. However, 
>> won't removing Primary Storage then re-adding it cause all data on 
>> the LUN's to be effectively wiped?
>>
>> How would you propose I avoid something like this in the future? A 
>> standby cluster?
>>
>> Thanks again for your help!
>>
>> Cheers,
>>
>> Matthew
>>
>>
>>
>> On 14 May 2012, at 21:12, "Matthew 
>> Hartmann"<mhartmann@tls.net<mailto:mhartmann@tls.net>>   wrote:
>>
>>
>> On 5/14/2012 3:35 PM, Geoff Higginbottom wrote:
>> Matthew,
>>
>> If I understand correctly, your storage is on a different IP Range to 
>> your Management Server and Hosts etc, if this is the case you need a 
>> Storage Network, which also needs to be accessible from the 
>> Management Server.  If no Storage Network is configured, CloudStack 
>> will use the Management Network when trying to access Storage.
>> Odd, did something change? The Management Server has never needed to 
>> access the storage network before. I have tried using XenServer name 
>> labels for the Storage Network, but to no avail. Why would the 
>> Management Server need to access my iSCSI SAN (Primary Storage)? I 
>> have confirmed my my Management Server can still access Secondary 
>> Storage (NFS Server). All devices have an FQDN which can be resolved 
>> from the XenServer and Management Server.
>>
>> Re http://bugs.cloudstack.org/browse/CS-14655 this is only required 
>> if you are using NIC Bonding and I do not believe you are
>> Yes, we are using bonding but we are not using CSP. The bug report 
>> refers to bonding and XenServer 6.0.x but it is not clear whether the 
>> steps outlined should be taken regardless of whether or not CSP is 
>> being used in ones setup.
>>
>> Cheers,
>>
>> Matthew
>>
>> Regards
>>
>> Geoff
>>
>>
>> -----Original Message-----
>> From: Matthew Hartmann [mailto:mhartmann@tls.net]
>> Sent: 14 May 2012 20:27
>> To: 
>> cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>
>> Subject: Re: Two Issues with XenServer 6.0.2 and CS 3.0.1
>>
>> Please see my responses inline, below:
>>
>> On 5/14/2012 3:18 PM, Geoff Higginbottom wrote:
>> Matthew,
>>
>> It looks like you have forgot to label your physical networks and map 
>> them within the Zone Setup process, this is vital if you want to use 
>> dedicated NICs for any of the four network types, ie Management, 
>> Guest, Public or Storage.
>> This was a pre-existing zone that was operating well. The XenServer 
>> pool freaked out and unraveled quicker than the fall of Rome. I 
>> performed a clean install on my XenServer's and am simply trying to 
>> re-add them back into the Pod. I'm only using one Node at this moment 
>> just so I can get things going. When I initially setup my XenServer's 
>> and the pool, I made copious note should something like this happen. 
>> Now that it has unfortunately happened, configuring my XenServer's 
>> the same as they were initially when everything was functional 
>> doesn't seem to have helped.
>>
>> Re the Cloud Supplemental Pack, this is only required if you are using
>> Security Groups with a  Basic Zone
>> Right, but is the patch required regardless of whether or not CSP is 
>> installed? I have experienced a similar issue and I am *not* using CSP.
>>
>> Cheers,
>>
>> Matthew
>>
>> Regards
>>
>> Geoff
>>
>>
>> -----Original Message-----
>> From: Matthew Hartmann [mailto:mhartmann@tls.net]
>> Sent: 14 May 2012 19:42
>> To: CloudStack user/admin discussions
>> Cc: CloudStack Devs
>> Subject: Fwd: Two Issues with XenServer 6.0.2 and CS 3.0.1
>>
>>
>> I forgot to include the following information that I found in the 
>> database after the host was added:
>>
>> mysql>     select * from host where id=22 \G;
>> [...]
>> private_ip_address: 10.0.0.16
>> private_netmask: 255.255.254.0
>> private_mac_address: e8:9a:8f:22:9b:c4
>> storage_ip_address: 10.0.0.16
>> storage_netmask: 255.255.254.0
>> storage_mac_address: e8:9a:8f:22:9b:c4
>> storage_ip_address_2: 10.0.0.16
>> storage_mac_address_2: e8:9a:8f:22:9b:c4
>> storage_netmask_2: 255.255.254.0
>> [...]
>>
>> That information is incorrect, as the storage_ip_address and
>> storage_ip_address_2 are two completely different subnets and 
>> CloudStack does not see these network devices.
>>
>> Thanks!
>>
>> Matthew
>>
>> -------- Original Message --------
>> Subject:        Two Issues with XenServer 6.0.2 and CS 3.0.1
>> Date:   Mon, 14 May 2012 14:36:57 -0400
>> From:   Matthew Hartmann<mhartmann@tls.net<mailto:mhartmann@tls.net>>
>> Organization:   TLS.NET<http://TLS.NET>, Inc.
>> To:     CloudStack user/admin discussions
>> <cloudstack-users@incubator.apache.org<mailto:cloudstack-users@incubator.apache.org>>

>>
>> CC:     CloudStack 
>> Devs<cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>>
>>
>>
>>
>> First of all, it is not very clear whether or not this patch is 
>> required for XenServer with regard to the Cloud Supplemental Pack:
>> http://bugs.cloudstack.org/browse/CS-14655
>>
>> Could someone shed some light on this?
>>
>> Secondly, I'm having an issue with CloudStack trying to use my 
>> "cloud-private" network for Public, Guest, Management and Storage.
>>
>> Here is what my CS 3.0.1 Management Server log is showing:
>>
>> [java] INFO [xen.resource.CitrixResourceBase] (http-8080-6:) Private
>> Network is cloud-private for host 10.0.0.16 [java] INFO
>> [xen.resource.CitrixResourceBase] (http-8080-6:) Guest Network is
>> cloud-private for host 10.0.0.16 [java] INFO
>> [xen.resource.CitrixResourceBase] (http-8080-6:) Public Network is
>> cloud-private for host 10.0.0.16 [java] INFO
>> [xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 1 is
>> cloud-private for host 10.0.0.16 [java] INFO
>> [xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 2 is
>> cloud-private for host 10.0.0.16
>>
>> On my XenServer 6.0.2 machine I have my "cloud-private" and 
>> "cloud-public" networks. In CS, I have my Storage network set to "Use 
>> default gateway". The XenServer has access to the Internet and can 
>> resolve the FQDN of all hosts, management servers and the SAN on the 
>> storage network. I can ping the IP's of the SAN on both Storage 
>> subnets, yet CloudStack reports after adding the host that it can't 
>> see the storage network; obviously because it is incorrectly 
>> discovering my networks.
>>
>> I have verified in the config in my database for my networks:
>>
>> mysql>     select * from physical_network_traffic_types;
>> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+

>>
>> | id | uuid                                 | physical_network_id |
>> traffic_type | xen_network_label | kvm_network_label |
>> vmware_network_label | simulator_network_label | ovm_network_label |
>> vlan |
>> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+

>>
>> |  1 | 3a368a1d-5d33-4e11-80ea-02da222d3930 |                   1 |
>> Management   | cloud-private     | NULL              |
>> NULL                 | NULL                    | NULL              | 
>> NULL |
>> |  2 | 3bf7b069-355e-4171-8ac5-c6f84777fc9d |                   2 |
>> Guest        | cloud-public      | NULL              |
>> NULL                 | NULL                    | NULL              | 
>> NULL |
>> |  3 | bb19c965-d726-451e-98fe-e0363c3bbd19 |                   2 |
>> Public       | cloud-public      | NULL              |
>> NULL                 | NULL                    | NULL              | 
>> NULL |
>> |  4 | 9df97cc6-f4e6-4d5f-b148-99f0dfc64dac |                   1 |
>> Storage      | NULL              | NULL              |
>> NULL                 | NULL                    | NULL              | 
>> NULL |
>> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+

>>
>> 4 rows in set (0.00 sec)
>>
>> Thoughts? Any assistance would be greatly appreciated!
>>
>> Cheers,
>>
>> Matthew
>>
>>
>> ShapeBlue provides a range of strategic and technical consulting 
>> services to help IT Service Providers and enterprises to build a true 
>> IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack 
>> technology, allows IT Service providers and enterprises to deliver 
>> true, utility based, IaaS to the customer without reengineering 
>> existing physical, virtual or storage layers.
>>
>> ________________________________
>>
>> This email and any attachments to it may be confidential and are 
>> intended solely for the use of the individual to whom it is 
>> addressed. Any views or opinions expressed are solely those of the 
>> author and do not necessarily represent those of Shape Blue Ltd. If 
>> you are not the intended recipient of this email, you must neither 
>> take any action based upon its contents, nor copy or show it to 
>> anyone. Please contact the sender if you believe you have received 
>> this email in error. Shape Blue Ltd is a company incorporated in 
>> England&     Wales.
>> ShapeBlue provides a range of strategic and technical consulting 
>> services to help IT Service Providers and enterprises to build a true 
>> IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack 
>> technology, allows IT Service providers and enterprises to deliver 
>> true, utility based, IaaS to the customer without reengineering 
>> existing physical, virtual or storage layers.
>>
>> ________________________________
>>
>> This email and any attachments to it may be confidential and are 
>> intended solely for the use of the individual to whom it is 
>> addressed. Any views or opinions expressed are solely those of the 
>> author and do not necessarily represent those of Shape Blue Ltd. If 
>> you are not the intended recipient of this email, you must neither 
>> take any action based upon its contents, nor copy or show it to 
>> anyone. Please contact the sender if you believe you have received 
>> this email in error. Shape Blue Ltd is a company incorporated in 
>> England&    Wales.
>>
>> ShapeBlue provides a range of strategic and technical consulting 
>> services to help IT Service Providers and enterprises to build a true 
>> IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack 
>> technology, allows IT Service providers and enterprises to deliver 
>> true, utility based, IaaS to the customer without reengineering 
>> existing physical, virtual or storage layers.
>>
>> ________________________________
>>
>> This email and any attachments to it may be confidential and are 
>> intended solely for the use of the individual to whom it is 
>> addressed. Any views or opinions expressed are solely those of the 
>> author and do not necessarily represent those of Shape Blue Ltd. If 
>> you are not the intended recipient of this email, you must neither 
>> take any action based upon its contents, nor copy or show it to 
>> anyone. Please contact the sender if you believe you have received 
>> this email in error. Shape Blue Ltd is a company incorporated in 
>> England&   Wales.
>>
>>
>>
>>
>>
>> ShapeBlue provides a range of strategic and technical consulting 
>> services to help IT Service Providers and enterprises to build a true 
>> IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack 
>> technology, allows IT Service providers and enterprises to deliver 
>> true, utility based, IaaS to the customer without reengineering 
>> existing physical, virtual or storage layers.
>>
>> ________________________________
>>
>> This email and any attachments to it may be confidential and are 
>> intended solely for the use of the individual to whom it is 
>> addressed. Any views or opinions expressed are solely those of the 
>> author and do not necessarily represent those of Shape Blue Ltd. If 
>> you are not the intended recipient of this email, you must neither 
>> take any action based upon its contents, nor copy or show it to 
>> anyone. Please contact the sender if you believe you have received 
>> this email in error. Shape Blue Ltd is a company incorporated in 
>> England&  Wales.
>>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message