cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From François Bousquet <Francois.Bousq...@radialpoint.com>
Subject Primary Storage Allocated over 100% ?!
Date Fri, 22 Aug 2014 01:56:36 GMT
Hello,

We do have a strange problem with our Primary Storage Allocated. It is now at 189%! (Storage
is at 30%). So we cannot deploy anymore, because of the pool.storage.allocated.capacity.disabledthreshold=0.85

CloudStack version is 4.2.1
Hypervisor is VMware ESXi 5.1 with 2 x 2 TB VMFS volume in cluster
storage.overprovisioning.factor=2

If we do a list capacity, the capacitytotal for Primary Storage Allocated (type=3) looks fine
at 8 TB (8795019280384) but the capacityused value totally doesn’t make sense.

We don’t know how this happened or how to fix it. In order to prove my point I did the sum
of all volumes ready/destroy (we do not have snapshots) and the total is a little more than
4 TB (see SQL query below).

However, storage pools are reported with wrong disk used value on storage_pool_view.  Content
of resource_count table makes more sense.

It looks like our 2 primary storage pools disk_used_capacity are not reporting the right value.
VMware reports then 30% full (same as CloudStack storage value).

Do you have any idea why my 2 storage pool are not reporting allocated disk properly?

I tried restarting CloudStack management server with no success.


> list capacity
count = 9
capacity:
+----------------+---------------+-------------+--------------------------------------+------+--------------+
|  capacityused  | capacitytotal | percentused |                zoneid                | type
|   zonename   |
+----------------+---------------+-------------+--------------------------------------+------+--------------+
|  287158829056  |  927422582784 |    30.96    | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  0
  | HypertecProd |
|     106500     |     839904    |    12.68    | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  1
  | HypertecProd |
| 16702508511567 | 8795019280384 |    189.91   | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  3
  | HypertecProd |
|       2        |      231      |     0.87    | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  4
  | HypertecProd |
|       6        |      201      |     2.99    | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  5
  | HypertecProd |
|       0        |      100      |      0      | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  7
  | HypertecProd |
|       58       |      883      |     6.57    | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  8
  | HypertecProd |
|  61694476288   | 3248203235328 |     1.9     | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  6
  | HypertecProd |
| 1338303840256  | 4397509640192 |    30.43    | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 |  2
  | HypertecProd |
+----------------+---------------+-------------+--------------------------------------+------+--------------+

>

mysql> select state, sum(size) from volumes group by state;
+----------+----------------+
| state    | sum(size)      |
+----------+----------------+
| Destroy  |  1030792151040 |
| Expunged | 22460481142784 |
| Ready    |  4797277143040 |
+----------+----------------+
3 rows in set (0.00 sec)

mysql>
mysql> select * from storage_pool_view;
+----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
| id | uuid                                 | name         | status | path               
| pool_type | host_address                        | created             | removed | capacity_bytes
| capacity_iops | scope | hypervisor | cluster_id | cluster_uuid | cluster_name | cluster_type
| data_center_id | data_center_uuid                     | data_center_name | data_center_type
| pod_id | pod_uuid | pod_name | tag          | disk_used_capacity | disk_reserved_capacity
| job_id | job_uuid | job_status | job_account_id |
+----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
|  1 | e1efbd9d-123b-350f-a99b-21fb4b6b38f1 | CLOUD-PROD01 | Up     | /CLOUD/CLOUD-PROD01
| VMFS      | VMFS datastore: /CLOUD/CLOUD-PROD01 | 2014-06-04 19:27:21 | NULL    |  2198754820096
|          NULL | ZONE  | VMware     |       NULL | NULL         | NULL         | NULL   
     |              1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd     | Advanced
        |   NULL | NULL     | NULL     | CLOUD-PROD01 |      9036192478890 |             
        0 |   NULL | NULL     |       NULL |           NULL |
|  4 | 89506344-2c37-3eb0-8360-449a923a12c8 | CLOUD-PROD02 | Up     | /CLOUD/CLOUD-PROD02
| VMFS      | VMFS datastore: /CLOUD/CLOUD-PROD02 | 2014-06-10 01:51:01 | NULL    |  2198754820096
|          NULL | ZONE  | VMware     |       NULL | NULL         | NULL         | NULL   
     |              1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd     | Advanced
        |   NULL | NULL     | NULL     | CLOUD-PROD02 |      7666316032677 |             
        0 |   NULL | NULL     |       NULL |           NULL |
+----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
2 rows in set (0.00 sec)

mysql> select * from resource_count where type='primary_storage';
+-----+------------+-----------+-----------------+---------------+
| id  | account_id | domain_id | type            | count         |
+-----+------------+-----------+-----------------+---------------+
|   7 |       NULL |         1 | primary_storage | 5817583534080 |
|  67 |          1 |      NULL | primary_storage |             0 |
| 103 |          2 |      NULL | primary_storage |   -2097152000 |
| 139 |          4 |      NULL | primary_storage |  214748364800 |
| 175 |          7 |      NULL | primary_storage | 2491081031680 |
| 211 |         10 |      NULL | primary_storage | 2093796556800 |
| 247 |         13 |      NULL | primary_storage | 1020054732800 |
+-----+------------+-----------+-----------------+---------------+
7 rows in set (0.00 sec)

Thanks


François Bousquet
NOC Team Lead
Office: 514-286-2636 #2337
Mobile: 514-654-8920
francois.bousquet@radialpoint.com




*********************************************************************** This e-mail and attachments
are confidential, legally privileged, may be subject to copyright and sent solely for the
attention of the addressee(s). Any unauthorized use or disclosure is prohibited. Statements
and opinions expressed in this e-mail may not represent those of Radialpoint. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Le contenu du présent courriel est confidentiel, privilégié et peut être soumis à des
droits d'auteur. Il est envoyé à l'intention exclusive de son ou de ses destinataires. Il
est interdit de l'utiliser ou de le divulguer sans autorisation. Les opinions exprimées dans
le présent courriel peuvent diverger de celles de Radialpoint.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message