cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geoff Higginbottom <geoff.higginbot...@shapeblue.com>
Subject RE: Primary Storage Allocated over 100% ?!
Date Fri, 22 Aug 2014 14:55:09 GMT
Hi Francois,

After upping your storage.overprovisioning.factor to 2, and restarted services, have you seen
any improvement, if not you may need to do the un-manage / manage Cluster trick to force CloudStack
to re-calculate the capacity and allocations etc

Regards

Geoff Higginbottom

D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581

geoff.higginbottom@shapeblue.com

-----Original Message-----
From: François Bousquet [mailto:Francois.Bousquet@radialpoint.com]
Sent: 22 August 2014 15:46
To: users@cloudstack.apache.org
Subject: RE: Primary Storage Allocated over 100% ?!

This installation is 4.2.1 and was never upgraded.

Now, we have set storage.overprovisioning.factor=2 in the global parameters, but I do not
see an storage overprovisioning factor at the cluster level. I only see the one for memory
and cpu.

-----Original Message-----
From: Geoff Higginbottom [mailto:geoff.higginbottom@shapeblue.com]
Sent: Friday, August 22, 2014 08:29
To: users@cloudstack.apache.org
Subject: RE: Primary Storage Allocated over 100% ?!

Hi Francois,

Have you updated to 4.2.1 recently?

4.2.1 introduced a lot of the features for 4.3, one of which was the addition of 'storage.overprovisioning.factor'
to each Cluster, where previously it was only a Global Settings

The update resets the Global Setting 'storage.overprovisioning.factor' to 1, then sets the
same value on each Cluster, even if previously it was set to a higher figure.

This value directly affects the allocated calculation and is vital if using NFS which automatically
thin provisions.

Please confirm what you current value is, at both Global and Cluster lever, and if it is 1,
suggest you set it to 4, then restart you cloudstack-management service on all management
servers.

We dealt with a similar issue recently via our CloudStack Support offering, and I recall we
also had to 'un-manage' then 're-manage' each Cluster for the new value to take effect.

Regards

Geoff Higginbottom

D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581

geoff.higginbottom@shapeblue.com

-----Original Message-----
From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
Sent: 22 August 2014 12:57
To: users@cloudstack.apache.org
Subject: Re: Primary Storage Allocated over 100% ?!

Hi Francois,

Could it be possible after a certain period metric inside cloudstack get updated to match
available resources? I'm wondering if it could be because of global settings like this one:

storage.stats.interval "The interval (in milliseconds) when storage stats (per host) are retrieved
from agents."  default=60000

Pierre-Luc



On Fri, Aug 22, 2014 at 6:53 AM, François Bousquet < Francois.Bousquet@radialpoint.com>
wrote:

> That may explain how I get there, but do you have a suggestion how to
> fix this issue if I can't find any volume big enough to justify all
> that allocated space?
>
> The sum of all the volumes is much lower than my 189% (15TB/8TB)
> primary allocated storage.
>
> I'm kind if stuck right now, not being able to fix the situation.
>
> > On Aug 22, 2014, at 0:33, "Prashant Kumar Mishra" <
> prashantkumar.mishra@citrix.com> wrote:
> >
> > Similar issue seen because of disk resize
> https://issues.apache.org/jira/browse/CLOUDSTACK-6797  .
> >
> > Thanks
> > prashant
> >
> > -----Original Message-----
> > From: François Bousquet [mailto:Francois.Bousquet@radialpoint.com]
> > Sent: Friday, August 22, 2014 7:27 AM
> > To: users@cloudstack.apache.org
> > Subject: Primary Storage Allocated over 100% ?!
> >
> > Hello,
> >
> > We do have a strange problem with our Primary Storage Allocated. It
> > is
> now at 189%! (Storage is at 30%). So we cannot deploy anymore, because
> of the pool.storage.allocated.capacity.disabledthreshold=0.85
> >
> > CloudStack version is 4.2.1
> > Hypervisor is VMware ESXi 5.1 with 2 x 2 TB VMFS volume in cluster
> > storage.overprovisioning.factor=2
> >
> > If we do a list capacity, the capacitytotal for Primary Storage
> Allocated (type=3) looks fine at 8 TB (8795019280384) but the
> capacityused value totally doesn’t make sense.
> >
> > We don’t know how this happened or how to fix it. In order to prove
> > my
> point I did the sum of all volumes ready/destroy (we do not have
> snapshots) and the total is a little more than 4 TB (see SQL query below).
> >
> > However, storage pools are reported with wrong disk used value on
> storage_pool_view.  Content of resource_count table makes more sense.
> >
> > It looks like our 2 primary storage pools disk_used_capacity are not
> reporting the right value. VMware reports then 30% full (same as
> CloudStack storage value).
> >
> > Do you have any idea why my 2 storage pool are not reporting
> > allocated
> disk properly?
> >
> > I tried restarting CloudStack management server with no success.
> >
> >
> >> list capacity
> > count = 9
> > capacity:
> >
> +----------------+---------------+-------------+--------------------------------------+------+--------------+
> > |  capacityused  | capacitytotal | percentused |                zoneid
>               | type |   zonename   |
> >
> +----------------+---------------+-------------+--------------------------------------+------+--------------+
> > |  287158829056  |  927422582784 |    30.96    |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  0   | HypertecProd |
> > |     106500     |     839904    |    12.68    |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  1   | HypertecProd |
> > | 16702508511567 | 8795019280384 |    189.91   |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  3   | HypertecProd |
> > |       2        |      231      |     0.87    |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  4   | HypertecProd |
> > |       6        |      201      |     2.99    |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  5   | HypertecProd |
> > |       0        |      100      |      0      |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  7   | HypertecProd |
> > |       58       |      883      |     6.57    |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  8   | HypertecProd |
> > |  61694476288   | 3248203235328 |     1.9     |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  6   | HypertecProd |
> > | 1338303840256  | 4397509640192 |    30.43    |
> 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  2   | HypertecProd |
> >
> +----------------+---------------+-------------+--------------------------------------+------+--------------+
> >
> >
> > mysql> select state, sum(size) from volumes group by state;
> > +----------+----------------+
> > | state    | sum(size)      |
> > +----------+----------------+
> > | Destroy  |  1030792151040 |
> > | Expunged | 22460481142784 |
> > | Ready    |  4797277143040 |
> > +----------+----------------+
> > 3 rows in set (0.00 sec)
> >
> > mysql>
> > mysql> select * from storage_pool_view;
> >
> +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
> > | id | uuid                                 | name         | status |
> path                | pool_type | host_address                        |
> created             | removed | capacity_bytes | capacity_iops | scope |
> hypervisor | cluster_id | cluster_uuid | cluster_name | cluster_type |
> data_center_id | data_center_uuid                     | data_center_name |
> data_center_type | pod_id | pod_uuid | pod_name | tag          |
> disk_used_capacity | disk_reserved_capacity | job_id | job_uuid |
> job_status | job_account_id |
> >
> +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
> > |  1 | e1efbd9d-123b-350f-a99b-21fb4b6b38f1 | CLOUD-PROD01 | Up     |
> /CLOUD/CLOUD-PROD01 | VMFS      | VMFS datastore: /CLOUD/CLOUD-PROD01 |
> 2014-06-04 19:27:21 | NULL    |  2198754820096 |          NULL | ZONE  |
> VMware     |       NULL | NULL         | NULL         | NULL         |
>         1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd     |
> Advanced         |   NULL | NULL     | NULL     | CLOUD-PROD01 |
> 9036192478890 |                      0 |   NULL | NULL     |       NULL |
>          NULL |
> > |  4 | 89506344-2c37-3eb0-8360-449a923a12c8 | CLOUD-PROD02 | Up     |
> /CLOUD/CLOUD-PROD02 | VMFS      | VMFS datastore: /CLOUD/CLOUD-PROD02 |
> 2014-06-10 01:51:01 | NULL    |  2198754820096 |          NULL | ZONE  |
> VMware     |       NULL | NULL         | NULL         | NULL         |
>         1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd     |
> Advanced         |   NULL | NULL     | NULL     | CLOUD-PROD02 |
> 7666316032677 |                      0 |   NULL | NULL     |       NULL |
>          NULL |
> >
> +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
> > 2 rows in set (0.00 sec)
> >
> > mysql> select * from resource_count where type='primary_storage';
> > +-----+------------+-----------+-----------------+---------------+
> > | id  | account_id | domain_id | type            | count         |
> > +-----+------------+-----------+-----------------+---------------+
> > |   7 |       NULL |         1 | primary_storage | 5817583534080 |
> > |  67 |          1 |      NULL | primary_storage |             0 |
> > | 103 |          2 |      NULL | primary_storage |   -2097152000 |
> > | 139 |          4 |      NULL | primary_storage |  214748364800 |
> > | 175 |          7 |      NULL | primary_storage | 2491081031680 |
> > | 211 |         10 |      NULL | primary_storage | 2093796556800 |
> > | 247 |         13 |      NULL | primary_storage | 1020054732800 |
> > +-----+------------+-----------+-----------------+---------------+
> > 7 rows in set (0.00 sec)
> >
> > Thanks
> >
> >
> > François Bousquet
> > NOC Team Lead
> > Office: 514-286-2636 #2337
> > Mobile: 514-654-8920
> > francois.bousquet@radialpoint.com
> >
> >
> >
> >
> > ********************************************************************
> > ***
> This e-mail and attachments are confidential, legally privileged, may
> be subject to copyright and sent solely for the attention of the addressee(s).
> Any unauthorized use or disclosure is prohibited. Statements and
> opinions expressed in this e-mail may not represent those of Radialpoint.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du
> présent courriel est confidentiel, privilégié et peut être soumis à
> des droits d'auteur. Il est envoyé à l'intention exclusive de son ou
> de ses destinataires. Il est interdit de l'utiliser ou de le divulguer
> sans autorisation. Les opinions exprimées dans le présent courriel
> peuvent diverger de celles de Radialpoint.
> **********************************************************************
> * This e-mail and attachments are confidential, legally privileged,
> may be subject to copyright and sent solely for the attention of the
> addressee(s).
> Any unauthorized use or disclosure is prohibited. Statements and
> opinions expressed in this e-mail may not represent those of Radialpoint.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du
> présent courriel est confidentiel, privilégié et peut être soumis à
> des droits d'auteur. Il est envoyé à l'intention exclusive de son ou
> de ses destinataires. Il est interdit de l'utiliser ou de le divulguer
> sans autorisation. Les opinions exprimées dans le présent courriel
> peuvent diverger de celles de Radialpoint.
>
Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.
*********************************************************************** This e-mail and attachments
are confidential, legally privileged, may be subject to copyright and sent solely for the
attention of the addressee(s). Any unauthorized use or disclosure is prohibited. Statements
and opinions expressed in this e-mail may not represent those of Radialpoint. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Le contenu du présent courriel est confidentiel, privilégié et peut être soumis à des
droits d'auteur. Il est envoyé à l'intention exclusive de son ou de ses destinataires. Il
est interdit de l'utiliser ou de le divulguer sans autorisation. Les opinions exprimées dans
le présent courriel peuvent diverger de celles de Radialpoint.
Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.
Mime
View raw message