cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Hartmann <mhartm...@tls.net>
Subject Re: Two Issues with XenServer 6.0.2 and CS 3.0.1
Date Wed, 16 May 2012 14:00:09 GMT
Well, I'm at the point I'm ready to remove Primary & Secondary Storage, 
however CloudStack refuses to remove them even though there are 0 hosts 
configured. CloudStack reports when trying to remove Primary Storage: 
"Failed to delete storage pool"

When trying to delete Secondary Storage CloudStack reports: "Can not 
delete this secondary storage due to there are still snapshots on it"

I disabled the Pod and Zone before attempting to remove Primary and 
Secondary Storage, as well as putting Primary Storage in Maintenance Mode.

Management Server log reports when trying to delete Secondary Storage:

2012-05-16 09:57:29,553 ERROR [cloud.api.ApiDispatcher] 
(catalina-exec-24:null) Exception while executing DeleteHostCmd:
com.cloud.utils.exception.CloudRuntimeException: Can not delete this 
secondary storage due to there are still snapshots on it

Management Server log reports when trying to delete Primary Storage:

2012-05-16 09:58:53,600 WARN  [cloud.storage.StorageManagerImpl] 
(catalina-exec-2:null) Cannot delete pool san-p1n1c1.iedc.cloud.tls.net 
as there are associated vols for this pool
2012-05-16 09:58:53,601 WARN  [cloud.api.ApiDispatcher] 
(catalina-exec-2:null) class com.cloud.api.ServerApiException : Failed 
to delete storage pool

I suspect manual scrubbing of the database is in order?

Cheers,

Matthew


On 5/14/2012 5:17 PM, Geoff Higginbottom wrote:
> Matthew,
>
> Removing and then re-adding Pri Storage will not wipe the LUNs and the existing VHD files
which are there, assuming the associated Guest Instances are still in the database should
also remap.
>
> How to avoid this in the future? It's pretty rare for a XenServer cluster to bomb out
in this way, but a 'standby cluster' would not really help as that would have its own Storage
Pool.  However having at least two Primary Storage pools in the Cluster (eg two iSCSI LUNs)
will allow you to migrate Guest Instances between them should you detect any problems.
>
> Also having multiple PODs and Clusters and forcing the distribution of Guest Instances
for a particular Account to be spread amongst them, rather than relying on the default 'Random'
allocation policies reduces risks in the event of any failures.
>
> If you do have data on the existing Primary Storage and want to be über safe (my preferred
stance) then copy the VHDs off first, you can always re-import them back into CloudStack via
Templates if you have to, but I assume as there are no clients on this system yet the LUNs
should be empty.
>
> Kind Regards
>
> Geoff Higginbottom
> CTO / Senior Consultant
> ShapeBlue Ltd
>
> geoff.higginbottom@shapeblue.com<mailto:geoff.higginbottom@shapeblue.com>  | www.shapeblue.com<http://www.shapeblue.com/>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
>
>
> On 14 May 2012, at 22:02, "Matthew Hartmann"<mhartmann@tls.net<mailto:mhartmann@tls.net>>
 wrote:
>
> On 5/14/2012 4:48 PM, Geoff Higginbottom wrote:
> I had assumed you had Primary and Secondary storage on different IP ranges to your Management
Server which does need to be able to access Secondary Storage but not Primary Storage.
>
> That assumption is correct, Primary Storage is on a different subnet than Secondary Storage.
Secondary Storage is sitting on the Management Network subnet.
>
>
> I now note your Primary Storage is iSCSI, and that you had a failure of all Hosts.
>
> When adding Storage and Hosts, you have to add Hosts, before you add Primary Storage,
and you have to add Primary Storage before Secondary Storage.
>
> This is all taken care of by the Add Zone Wizard, but if you have a failure of all Hosts
in a Zone then you need to remove both Primary and Secondary Storage, then add a Host, then
add Primary Storage, then add Secondary storage.
>
> Templates, ISOs etc that are on Secondary will disappear from the GUI, but don't worry,
they will be detected by the SSVM once it comes online, and they will then reappear in the
GUI.
>
> If this is a simple test system then it may be easier to simply drop the DB Schema, and
recreate it from scratch, or create a new Zone and let the wizard take care of everything.
>
> This actually was a system we had *just* placed into production. We luckily did not have
any clients setup on this cloud yet. However, won't removing Primary Storage then re-adding
it cause all data on the LUN's to be effectively wiped?
>
> How would you propose I avoid something like this in the future? A standby cluster?
>
> Thanks again for your help!
>
> Cheers,
>
> Matthew
>
>
>
> On 14 May 2012, at 21:12, "Matthew Hartmann"<mhartmann@tls.net<mailto:mhartmann@tls.net>>
  wrote:
>
>
> On 5/14/2012 3:35 PM, Geoff Higginbottom wrote:
> Matthew,
>
> If I understand correctly, your storage is on a different IP Range to your Management
Server and Hosts etc, if this is the case you need a Storage Network, which also needs to
be accessible from the Management Server.  If no Storage Network is configured, CloudStack
will use the Management Network when trying to access Storage.
> Odd, did something change? The Management Server has never needed to access the storage
network before. I have tried using XenServer name labels for the Storage Network, but to no
avail. Why would the Management Server need to access my iSCSI SAN (Primary Storage)? I have
confirmed my my Management Server can still access Secondary Storage (NFS Server). All devices
have an FQDN which can be resolved from the XenServer and Management Server.
>
> Re http://bugs.cloudstack.org/browse/CS-14655 this is only required if you are using
NIC Bonding and I do not believe you are
> Yes, we are using bonding but we are not using CSP. The bug report refers to bonding
and XenServer 6.0.x but it is not clear whether the steps outlined should be taken regardless
of whether or not CSP is being used in ones setup.
>
> Cheers,
>
> Matthew
>
> Regards
>
> Geoff
>
>
> -----Original Message-----
> From: Matthew Hartmann [mailto:mhartmann@tls.net]
> Sent: 14 May 2012 20:27
> To: cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>
> Subject: Re: Two Issues with XenServer 6.0.2 and CS 3.0.1
>
> Please see my responses inline, below:
>
> On 5/14/2012 3:18 PM, Geoff Higginbottom wrote:
> Matthew,
>
> It looks like you have forgot to label your physical networks and map them within the
Zone Setup process, this is vital if you want to use dedicated NICs for any of the four network
types, ie Management, Guest, Public or Storage.
> This was a pre-existing zone that was operating well. The XenServer pool freaked out
and unraveled quicker than the fall of Rome. I performed a clean install on my XenServer's
and am simply trying to re-add them back into the Pod. I'm only using one Node at this moment
just so I can get things going. When I initially setup my XenServer's and the pool, I made
copious note should something like this happen. Now that it has unfortunately happened, configuring
my XenServer's the same as they were initially when everything was functional doesn't seem
to have helped.
>
> Re the Cloud Supplemental Pack, this is only required if you are using
> Security Groups with a  Basic Zone
> Right, but is the patch required regardless of whether or not CSP is installed? I have
experienced a similar issue and I am *not* using CSP.
>
> Cheers,
>
> Matthew
>
> Regards
>
> Geoff
>
>
> -----Original Message-----
> From: Matthew Hartmann [mailto:mhartmann@tls.net]
> Sent: 14 May 2012 19:42
> To: CloudStack user/admin discussions
> Cc: CloudStack Devs
> Subject: Fwd: Two Issues with XenServer 6.0.2 and CS 3.0.1
>
>
> I forgot to include the following information that I found in the database after the
host was added:
>
> mysql>     select * from host where id=22 \G;
> [...]
> private_ip_address: 10.0.0.16
> private_netmask: 255.255.254.0
> private_mac_address: e8:9a:8f:22:9b:c4
> storage_ip_address: 10.0.0.16
> storage_netmask: 255.255.254.0
> storage_mac_address: e8:9a:8f:22:9b:c4
> storage_ip_address_2: 10.0.0.16
> storage_mac_address_2: e8:9a:8f:22:9b:c4
> storage_netmask_2: 255.255.254.0
> [...]
>
> That information is incorrect, as the storage_ip_address and
> storage_ip_address_2 are two completely different subnets and CloudStack does not see
these network devices.
>
> Thanks!
>
> Matthew
>
> -------- Original Message --------
> Subject:        Two Issues with XenServer 6.0.2 and CS 3.0.1
> Date:   Mon, 14 May 2012 14:36:57 -0400
> From:   Matthew Hartmann<mhartmann@tls.net<mailto:mhartmann@tls.net>>
> Organization:   TLS.NET<http://TLS.NET>, Inc.
> To:     CloudStack user/admin discussions
> <cloudstack-users@incubator.apache.org<mailto:cloudstack-users@incubator.apache.org>>
> CC:     CloudStack Devs<cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>>
>
>
>
> First of all, it is not very clear whether or not this patch is required for XenServer
with regard to the Cloud Supplemental Pack:
> http://bugs.cloudstack.org/browse/CS-14655
>
> Could someone shed some light on this?
>
> Secondly, I'm having an issue with CloudStack trying to use my "cloud-private" network
for Public, Guest, Management and Storage.
>
> Here is what my CS 3.0.1 Management Server log is showing:
>
> [java] INFO [xen.resource.CitrixResourceBase] (http-8080-6:) Private
> Network is cloud-private for host 10.0.0.16 [java] INFO
> [xen.resource.CitrixResourceBase] (http-8080-6:) Guest Network is
> cloud-private for host 10.0.0.16 [java] INFO
> [xen.resource.CitrixResourceBase] (http-8080-6:) Public Network is
> cloud-private for host 10.0.0.16 [java] INFO
> [xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 1 is
> cloud-private for host 10.0.0.16 [java] INFO
> [xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 2 is
> cloud-private for host 10.0.0.16
>
> On my XenServer 6.0.2 machine I have my "cloud-private" and "cloud-public" networks.
In CS, I have my Storage network set to "Use default gateway". The XenServer has access to
the Internet and can resolve the FQDN of all hosts, management servers and the SAN on the
storage network. I can ping the IP's of the SAN on both Storage subnets, yet CloudStack reports
after adding the host that it can't see the storage network; obviously because it is incorrectly
discovering my networks.
>
> I have verified in the config in my database for my networks:
>
> mysql>     select * from physical_network_traffic_types;
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+
> | id | uuid                                 | physical_network_id |
> traffic_type | xen_network_label | kvm_network_label |
> vmware_network_label | simulator_network_label | ovm_network_label |
> vlan |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+
> |  1 | 3a368a1d-5d33-4e11-80ea-02da222d3930 |                   1 |
> Management   | cloud-private     | NULL              |
> NULL                 | NULL                    | NULL              | NULL |
> |  2 | 3bf7b069-355e-4171-8ac5-c6f84777fc9d |                   2 |
> Guest        | cloud-public      | NULL              |
> NULL                 | NULL                    | NULL              | NULL |
> |  3 | bb19c965-d726-451e-98fe-e0363c3bbd19 |                   2 |
> Public       | cloud-public      | NULL              |
> NULL                 | NULL                    | NULL              | NULL |
> |  4 | 9df97cc6-f4e6-4d5f-b148-99f0dfc64dac |                   1 |
> Storage      | NULL              | NULL              |
> NULL                 | NULL                    | NULL              | NULL |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+
> 4 rows in set (0.00 sec)
>
> Thoughts? Any assistance would be greatly appreciated!
>
> Cheers,
>
> Matthew
>
>
> ShapeBlue provides a range of strategic and technical consulting services to help IT
Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise,
combined with CloudStack technology, allows IT Service providers and enterprises to deliver
true, utility based, IaaS to the customer without reengineering existing physical, virtual
or storage layers.
>
> ________________________________
>
> This email and any attachments to it may be confidential and are intended solely for
the use of the individual to whom it is addressed. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not
the intended recipient of this email, you must neither take any action based upon its contents,
nor copy or show it to anyone. Please contact the sender if you believe you have received
this email in error. Shape Blue Ltd is a company incorporated in England&     Wales.
> ShapeBlue provides a range of strategic and technical consulting services to help IT
Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise,
combined with CloudStack technology, allows IT Service providers and enterprises to deliver
true, utility based, IaaS to the customer without reengineering existing physical, virtual
or storage layers.
>
> ________________________________
>
> This email and any attachments to it may be confidential and are intended solely for
the use of the individual to whom it is addressed. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not
the intended recipient of this email, you must neither take any action based upon its contents,
nor copy or show it to anyone. Please contact the sender if you believe you have received
this email in error. Shape Blue Ltd is a company incorporated in England&    Wales.
>
> ShapeBlue provides a range of strategic and technical consulting services to help IT
Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise,
combined with CloudStack technology, allows IT Service providers and enterprises to deliver
true, utility based, IaaS to the customer without reengineering existing physical, virtual
or storage layers.
>
> ________________________________
>
> This email and any attachments to it may be confidential and are intended solely for
the use of the individual to whom it is addressed. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not
the intended recipient of this email, you must neither take any action based upon its contents,
nor copy or show it to anyone. Please contact the sender if you believe you have received
this email in error. Shape Blue Ltd is a company incorporated in England&   Wales.
>
>
>
>
>
> ShapeBlue provides a range of strategic and technical consulting services to help IT
Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise,
combined with CloudStack technology, allows IT Service providers and enterprises to deliver
true, utility based, IaaS to the customer without reengineering existing physical, virtual
or storage layers.
>
> ________________________________
>
> This email and any attachments to it may be confidential and are intended solely for
the use of the individual to whom it is addressed. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not
the intended recipient of this email, you must neither take any action based upon its contents,
nor copy or show it to anyone. Please contact the sender if you believe you have received
this email in error. Shape Blue Ltd is a company incorporated in England&  Wales.
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message