cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From he...@powershield.ru
Subject Re: VR starting problem
Date Tue, 07 Nov 2017 08:32:32 GMT
Good morning for everyone,

we still have the problem, anyone have any ideas?

Primary storage is presetup'ed and can see it in GUI. In logs, we found 
this:

StorageUnavailableException: Resource [StoragePool:6] is unreachable

We can't start systemvm's.

All logs:

https://pastebin.com/JU0qr69V
https://pastebin.com/A5Z4zA2U


Paul Angus писал 2017-11-07 00:18:
> Yes, I see that, but as the message is still the same, you either
> changed a similar but different setting (there are a number of
> similarly named global settings) or you did not restart all of your
> management servers.
> 
> 
> 
> Kind regards,
> 
> Paul Angus
> 
> paul.angus@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 
> 
> 
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 21:15
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
> 
> As i wrote before, i already did it before first my e-mail here, but
> without luck:)
> 
> 
> 
> Paul Angus писал 2017-11-07 00:12:
>> Ideally we'd need to start the log even earlier, but I can still see:
>> Insufficient space on pool: 6 since its usage percentage:
>> 0.8669012505498059 has crossed the
>> pool.storage.capacity.disablethreshold: 0.85
>> 
>> If you increased pool.storage.capacity.disablethreshold to above 087,
>> then you need to restart your management server(s) for it to take
>> effect.
>> 
>> It would be better though, if you add another primary storage pool, or
>> increase the storage in the existing pool. As increasing the threshold
>> isn't a long term solution.
>> 
>> 
>> 
>> 
>> 
>> 
>> Kind regards,
>> 
>> Paul Angus
>> 
>> paul.angus@shapeblue.com 
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>> 
>> 
>> 
>> 
>> -----Original Message-----
>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>> Sent: 06 November 2017 20:53
>> To: users@cloudstack.apache.org
>> Subject: Re: VR starting problem
>> 
>> Hello,
>> 
>> Here is some of log.
>> 
>> https://pastebin.com/JU0qr69V
>> 
>> Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
>> 
>> As you can see, datacenter (zone) showing, but can't see pod and
>> cluster....
>> 
>> 
>> Paul Angus писал 2017-11-06 22:44:
>>> Please could you use something like pastebin (https://pastebin.com/)
>>> to paste larger sections of your logs.
>>> The section you have pasted is more likely from XenServer trying to
>>> remove a disk image that was never created in the first place.  It 
>>> may
>>> not be the root cause.
>>> 
>>> Kind regards,
>>> 
>>> Paul Angus
>>> 
>>> paul.angus@shapeblue.com 
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> @shapeblue
>>> 
>>> 
>>> 
>>> 
>>> -----Original Message-----
>>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>>> Sent: 06 November 2017 18:14
>>> To: users@cloudstack.apache.org
>>> Subject: Re: VR starting problem
>>> 
>>> We deleted system VM's and also, can't start.
>>> 
>>> Now seeing the error:
>>> 
>>> 
>>> INFO  [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
>>> (logid:66199d2c) Unable to contact resource.
>>> com.cloud.exception.StorageUnavailableException: Resource
>>> [StoragePool:6] is unreachable: Unable to create
>>> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>>> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
>>> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
>>> you supplied was invalid.
>>> 
>>> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>> 2017-11-06 20:16:49,501 WARN  [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create 
>>> volume;
>>> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
>>> Disk:
>>> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> hello@powershield.ru писал 2017-11-06 20:48:
>>>> Changed few hours ago, and anyway still the same problem.
>>>> 
>>>> 
>>>> 
>>>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>>>> threshold to 0.98 and go ahead.
>>>>> 
>>>>> 7 нояб. 2017 г. 0:22 пользователь <hello@powershield.ru>
написал:
>>>>> 
>>>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>>>> the problem is same. All VM's working but if client restart - they
>>>>>> goes down, with the same error with insufficient storage.
>>>>>> 
>>>>>> 
>>>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>>> 
>>>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>>>> maintanence mode?
>>>>>>> 
>>>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev"
<
>>>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>>> 
>>>>>>> Hi. Try to find in early logs what exactly resource is
>>>>>>> insufficient.
>>>>>>> Try
>>>>>>>> to play with threshold of it.
>>>>>>>> 
>>>>>>>> 6 нояб. 2017 г. 21:57 пользователь <hello@powershield.ru>
>>>>>>>> написал:
>>>>>>>> 
>>>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>>> 
>>>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode:
530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17]
is
>>>>>>>>> unreachable: Host 17: Unable to start instance due to
Unable to
>>>>>>>>> start
>>>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error
in
>>>>>>>>> finalizeStart, not retrying"}
>>>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode:
530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start
a VM
>>>>>>>>> due to insufficient capacity"}
>>>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode:
530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource
>>>>>>>>> [DataCenter:2]
>>>>>>>>> is
>>>>>>>>> unreachable: Unable to reboot domR, it is not in right
state
>>>>>>>>> Stopped"}
>>>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode:
530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create
a
>>>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>>> 
>>>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>>> 
>>>>>>>>> Primary Storage Used
>>>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>>> 
>>>>>>>>> Primary Storage Allocated
>>>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>>> 
>>>>>>>>> Memory Allocated
>>>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>>> 
>>>>>>>>> CPU Allocated
>>>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>>> 
>>>>>>>>> Shared Network IPs
>>>>>>>>> Allocated: 34 / 253
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>>> 
>>>>>>>>> Hi, you should dive into management server logs to find

>>>>>>>>> relevant
>>>>>>>>>> messages.
>>>>>>>>>> Otherwise it's impossible to predict what's wrong
with the
>>>>>>>>>> system.
>>>>>>>>>> Check
>>>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>>> 
>>>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь
<hello@powershield.ru>
>>>>>>>>>> написал:
>>>>>>>>>> 
>>>>>>>>>> Hello,
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ACS: 4.8.0
>>>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> We have got some problem with additional space
for our 
>>>>>>>>>>> client.
>>>>>>>>>>> 
>>>>>>>>>>> We are created new Disk Offering, then we tryed
to attach for
>>>>>>>>>>> our client, but got error: Unable to find suitable
primary
>>>>>>>>>>> storage when creating volume
>>>>>>>>>>> 
>>>>>>>>>>> We checked datadisk and we see, there is not
hypervisor
>>>>>>>>>>> (should
>>>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>>> 
>>>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>>> 
>>>>>>>>>>> After it, we restarted VR and he can't start.
He started and
>>>>>>>>>>> again goes to offline. We tryed to check from
XenCenter and
>>>>>>>>>>> got
>>>>>>>>>>> this reason:
>>>>>>>>>>> apache2:
>>>>>>>>>>> Could not reliably determine the server's fully
qualified
>>>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>>> 
>>>>>>>>>>> We delete VR and now, we can not start it manually
and he is
>>>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 

Mime
View raw message