cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuri Kogun <yko...@outlook.com>
Subject RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
Date Tue, 27 Jan 2015 09:22:40 GMT
Thank you, Somesh 
Will try to upgrade.

Best regards 
Yuri 

----------------------------------------
> From: Somesh.Naidu@citrix.com
> To: users@cloudstack.apache.org
> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
> Date: Mon, 26 Jan 2015 18:49:54 +0000
>
>
> Yes, XS will have the correct view since it is the one that's managing the memory. The
problem is the way CS calculates the memory overhead. I don't think there is much we can do
in terms of configuration.
>
> This is how CS arrives at total ram:
> ram = (long)((ram - dom0Ram - _xsMemoryUsed) * _xsVirtualizationFactor);
> In 4.3.x we derive dom0Ram from dom0's memory-dynamic-max, [dom0Ram = vm.getMemoryStaticMax(conn);].
> The fix was to derive it from dom0's memory-static-max, [dom0Ram = vm.getMemoryStaticMax(conn);].
>
> I see the fix in 4.4.x. So you may want to upgrade to that version to fix this issue.
>
> Regards,
> Somesh
>
>
> -----Original Message-----
> From: Yuri Kogun [mailto:ykogun@outlook.com]
> Sent: Monday, January 26, 2015 11:10 AM
> To: users@cloudstack.apache.org
> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
>
> Hi Somesh,
> Thank you for the reply, yes I found the issue in the Jira.
>
> The affected host was running about 31 VM's and we don't use memory over provisioning
at all. Unfortunately cloudstack fails to restart he VM and it is a big issue for us as the
restart is a part of a nightly job and we pick it up only in the morning. The only way for
us address the issue is to remove 'active' tag from the host in question to stop it from being
used for starting new Vm's.
> I also noticed that the available memory is reported correctly in the xen server console,
but not inside cloudstack in the management log.
>
>
> Best regards
> Yuri
>
> ----------------------------------------
>> From: Somesh.Naidu@citrix.com
>> To: users@cloudstack.apache.org
>> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
>> Date: Mon, 26 Jan 2015 15:56:01 +0000
>>
>> Yes, seen this quite a few times. I believe you already found CLOUDSTACK-2344.
>>
>> Basically, the issue happens when cloudstack's view of available memory is out of
sync (more) than that of XS. This could happen due to incorrect calculation of memory overhead
and Dom0 memory. It is also possible that your memory overprovisioning value is set too high
so please verify that. You might also want to check how many VMs were running on that particular
host when this error is thrown. When a host is running too many VMs (excess of 60) then there
is a possibility for such issues.
>>
>> Having said that, I believe there are subsequent attempts by cloudstack to start
the VM on other hosts and the VM eventually starts. If not then we may be looking at a potential
defect.
>>
>> Regards,
>> Somesh
>>
>>
>> -----Original Message-----
>> From: Yuri Kogun [mailto:ykogun@outlook.com]
>> Sent: Monday, January 26, 2015 10:26 AM
>> To: users@cloudstack.apache.org
>> Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
>>
>> Hi,
>> I wonder if somebody experienced a similar issue. We have a very busy dev cloudstack
installation with 10 hosts, 3 clusters and more than 300 user VM's running across the clusters.
The cpu over-provisioning is set to 3. >From time to time we are getting the following
an error when starting the VM.
>>
>> 2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-578a4e5d)
Task failed! Task record: uuid: ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
>> nameLabel: Async.VM.start_on
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Jan 26 01:45:24 GMT 2015
>> finished: Mon Jan 26 01:45:24 GMT 2015
>> status: failure
>> residentOn: com.xensource.xenapi.Host@fcaebca8
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>>
>>
>> I traced the job executor for the VM and it looks like the process have reported
that the host have enough RAM 2060627968 to start the VM, which requestes 1572864000 but for
some reason the command failed on the Xen server.
>>
>> 2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) Deployment found - P0=VM[User|i-3-160816-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT-->Pool(30))]
>> 2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) VM state transitted from :Starting to Starting with event:
OperationRetryvm's original host id: null new host id: 35 host id before state transition:
null
>> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) Hosts's actual total CPU: 44688 and CPU after applying
overprovisioning: 58094
>> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) We are allocating VM, increasing the used capacity of
this host:35
>> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) Current Used CPU: 52500 , Free CPU:5594 ,Requested CPU:
1500
>> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) Current Used RAM: 61220061184 , Free RAM:2060627968 ,Requested
RAM: 1572864000
>> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) CPU STATS after allocation: for host: 35, old used: 52500,
old reserved: 0, actual total: 44688, total with o
>> verprovisioning: 58094; new used:54000, reserved:0; requested cpu:1500,alloc_from_last:false
>> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-c47b7a2f
job-205846/job-205847 ctx-ea6dc373) RAM STATS after allocation: for host: 35, old used: 61220061184,
old reserved: 0, total: 63280689152; new use
>> d: 62792925184, reserved: 0; requested mem: 1572864000,alloc_from_last:false
>>
>>
>> Please let me know if somebody had a similar problem and managed to fix it. We are
running xenserver6.2 with cloudstack 4.3.0.1
>>
>> Bast regards
>> Yuri
>
 		 	   		  
Mime
View raw message