cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ilya <ilya.mailing.li...@gmail.com>
Subject Re: KVM CPU Overprovisioning issues.
Date Sat, 18 Apr 2015 00:26:53 GMT
When the deployment goes through, the allocation placement algorithm 
will tell you if cpu over provisioning worked. You can see it in 
management-server.log file..

Here is an easy way to extract useful info for failed vm deployment. 
After deployment failed, grep for cloudstack internal name of the VM and 
note the job id (i see in your log it was job-7312) in 
management-server.log . Once you have the job id, grep 'job-xxxx' 
management-server.log. Post the content here... I do see you post it the 
log below, it appears we are missing some info.

You can override CPU overprovisioning values in global settings as well 
as cluster settings.

If you are enable CPU reservations, than it probably wont work, because 
you trying to reserve more than whats available.

What version of ACS you have and what hypervisor?

On 4/17/15 2:01 PM, Grayson Head wrote:
> I'm trying to figure out how CPU over-provisioning works in
> cloudstack. It seems that no matter how much I change the
> over-provisioning factor value in global settings, I cannot
> over-provision CPU allocation on the hosts. I suspect that I'm just
> having a lack of understanding on my part, but it seems like I should
> be able to increase the number of VMs I can deploy by increasing the
> CPU over-provisioning value (given that I'm not close to any other
> resource limits). But this doesn't seem to be the case in practice.
>
> https://screencloud.net/v/l1X7
>
> https://screencloud.net/v/9ifx
>
> Error message when trying to create additional machine via API.
>
> grayson@gworkstation:~$ cloudmonkey deploy virtualmachine
> zoneid=a4c20770-f17e-425e-bb21-413dd610b975
> serviceofferingid=4b391121-49c7-4434-883d-ffe569f63a3b
> templateid=e5701f73-b0c3-4dd7-8b2d-b8bf78875a85 keypair=key1
> networkids=86e6ded0-6155-449b-a4bd-03411bf0a9cd name=tvmsman1
> displayname &
> [1] 3907
> Async job 09148c21-3c42-4334-93b2-428b48fb1fa2 failed
> Error 533, Unable to create a deployment for VM[User|i-15-1151-VM]
>   accountid = a4e6eaed-9b06-4806-98f9-4fcc781b773f
> cmd = org.apache.cloudstack.api.command.user.vm.DeployVMCmd
> created = 2015-04-17T13:46:22-0700
> jobid = 09148c21-3c42-4334-93b2-428b48fb1fa2
> jobprocstatus = 0
> jobresult:
> errorcode = 533
> errortext = Unable to create a deployment for VM[User|i-15-1151-VM]
> jobresultcode = 530
> jobresulttype = object
> jobstatus = 2
> userid = c7a0889e-fcec-4095-9d07-d29ea3c1e5d9
>
>
> (local) mycloudmonkey> list hosts
> filter=name,clustername,cpuallocated,cpunumber,cpusockets,cpuspeed,cpuused,cpuwithoverprovisioning,
> name = v-78-VM
> ================================================================================
> name = s-871-VM
> ================================================================================
> name = cl2hyper2
> clustername = CL2
> cpuallocated = 100.3%
> cpunumber = 8
> cpusockets = 2
> cpuspeed = 1994
> cpuused = 1.2%
> cpuwithoverprovisioning = 63808.0
> ================================================================================
> name = v-1-VM
> ================================================================================
> name = cl2hyper1
> clustername = CL2
> cpuallocated = 81.45%
> cpunumber = 8
> cpusockets = 2
> cpuspeed = 1995
> cpuused = 0.6%
> cpuwithoverprovisioning = 63840.0
> ================================================================================
>
>
>
> 2015-04-17 13:46:22,517 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-25:ctx-7b0eb43a ctx-751b2eca ctx-5c8a75fc) submit async
> job-7312, details: AsyncJobVO {id:7312, userId: 15, accountId: 15,
> instanceType: VirtualMachine, instanceId: 1151, cmd:
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo:
> {"serviceofferingid":"4b391121-49c7-4434-883d-ffe569f63a3b","cmdEventType":"VM.CREATE","ctxUserId":"15","zoneid":"a4c20770-f17e-425e-bb21-413dd610b975","signatureversion":"3","httpmethod":"GET","templateid":"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85","networkids":"86e6ded0-6155-449b-a4bd-03411bf0a9cd","apiKey":"wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw","id":"1151","response":"json","ctxDetails":"{\"com.cloud.template.VirtualMachineTemplate\":\"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85\",\"com.cloud.vm.VirtualMachine\":\"6b3ace85-8e4c-4467-afdf-b511136ca6b1\",\"com.cloud.network.Network\":\"86e6ded0-6155-449b-a4bd-03411bf0a9cd\",\"com.cloud.dc.DataCenter\":\"a4c20770-f17e-425e-bb21-413dd610b975\",\"com.cloud.offering.ServiceOffering\":\"4b391121-49c7-4434-883d-ffe569f63a3b\"}","expires":"2015-04-17T20:56:22+0000","keypair":"key1","name":"tvmsman1","uuid":"6b3ace85-8e4c-4467-afdf-b511136ca6b1","ctxAccountId":"15","ctxStartEventId":"11922","signature":"8Ebg1+qjfnk5nv7xz1GtdSju2ws\u003d"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345050015385, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2015-04-17 13:46:22,518 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-16:ctx-e039eccb job-7312) Add job-7312 into job
> monitoring
> 2015-04-17 13:46:22,518 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312) Executing AsyncJobVO
> {id:7312, userId: 15, accountId: 15, instanceType: VirtualMachine,
> instanceId: 1151, cmd:
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo:
> {"serviceofferingid":"4b391121-49c7-4434-883d-ffe569f63a3b","cmdEventType":"VM.CREATE","ctxUserId":"15","zoneid":"a4c20770-f17e-425e-bb21-413dd610b975","signatureversion":"3","httpmethod":"GET","templateid":"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85","networkids":"86e6ded0-6155-449b-a4bd-03411bf0a9cd","apiKey":"wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw","id":"1151","response":"json","ctxDetails":"{\"com.cloud.template.VirtualMachineTemplate\":\"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85\",\"com.cloud.vm.VirtualMachine\":\"6b3ace85-8e4c-4467-afdf-b511136ca6b1\",\"com.cloud.network.Network\":\"86e6ded0-6155-449b-a4bd-03411bf0a9cd\",\"com.cloud.dc.DataCenter\":\"a4c20770-f17e-425e-bb21-413dd610b975\",\"com.cloud.offering.ServiceOffering\":\"4b391121-49c7-4434-883d-ffe569f63a3b\"}","expires":"2015-04-17T20:56:22+0000","keypair":"key1","name":"tvmsman1","uuid":"6b3ace85-8e4c-4467-afdf-b511136ca6b1","ctxAccountId":"15","ctxStartEventId":"11922","signature":"8Ebg1+qjfnk5nv7xz1GtdSju2ws\u003d"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345050015385, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2015-04-17 13:46:22,519 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-25:ctx-7b0eb43a ctx-751b2eca ctx-5c8a75fc) ===END===
> 10.101.1.28 -- GET
> networkids=86e6ded0-6155-449b-a4bd-03411bf0a9cd&apiKey=wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw&name=tvmsman1&serviceofferingid=4b391121-49c7-4434-883d-ffe569f63a3b&expires=2015-04-17T20%3A56%3A22%2B0000&signatureversion=3&zoneid=a4c20770-f17e-425e-bb21-413dd610b975&keypair=key1&command=deployVirtualMachine&templateid=e5701f73-b0c3-4dd7-8b2d-b8bf78875a85&signature=8Ebg1%2Bqjfnk5nv7xz1GtdSju2ws%3D&response=json
> 2015-04-17 13:46:22,522 DEBUG [o.a.c.a.BaseCmd]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
> paremeter displayvm as the caller is not authorized to pass it in
> 2015-04-17 13:46:22,522 DEBUG [o.a.c.a.BaseCmd]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
> paremeter deploymentplanner as the caller is not authorized to pass it
> in
> 2015-04-17 13:46:22,539 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
> Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] granted to
> Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
> 2015-04-17 13:46:22,552 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
> Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] granted to
> Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
> 2015-04-17 13:46:22,565 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
> Tmpl[222-QCOW2-222-3-98769ef0-8ddf-300a-874f-0e9320ad02c9 granted to
> Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
> 2015-04-17 13:46:22,565 DEBUG [o.a.c.a.BaseCmd]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
> paremeter displayvm as the caller is not authorized to pass it in
> 2015-04-17 13:46:22,565 DEBUG [o.a.c.a.BaseCmd]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
> paremeter deploymentplanner as the caller is not authorized to pass it
> in
> 2015-04-17 13:46:22,566 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Received
> unknown parameters for command deployVirtualMachine. Unknown
> parameters : signatureversion expires
> 2015-04-17 13:46:22,588 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
> VM[User|i-15-1151-VM] granted to
> Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
> 2015-04-17 13:46:22,614 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Service
> SecurityGroup is not supported in the network id=243
> 2015-04-17 13:46:22,621 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Service
> SecurityGroup is not supported in the network id=243
> 2015-04-17 13:46:22,658 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Deploy avoids
> pods: [], clusters: [], hosts: []
> 2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb)
> DeploymentPlanner allocation algorithm:
> com.cloud.deploy.FirstFitPlanner@6ba48686
> 2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Trying to
> allocate a host and storage pools from dc:1, pod:null,cluster:null,
> requested cpu: 500, requested ram: 536870912
> 2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Is ROOT
> volume READY (pool already allocated)?: No
> 2015-04-17 13:46:22,660 DEBUG [c.c.d.FirstFitPlanner]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Searching all
> possible resources under this Zone: 1
> 2015-04-17 13:46:22,662 DEBUG [c.c.d.FirstFitPlanner]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Listing
> clusters in order of aggregate capacity, that have (atleast one host
> with) enough CPU and RAM capacity under this Zone: 1
> 2015-04-17 13:46:22,668 DEBUG [c.c.d.FirstFitPlanner]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Removing from
> the clusterId list these clusters from avoid set: []
> 2015-04-17 13:46:22,678 DEBUG [c.c.d.FirstFitPlanner]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Cannot
> allocate cluster list [2] for vm creation since their allocated
> percentage crosses the disable capacity threshold defined at each
> cluster/ at global value for capacity Type : 1, skipping these
> clusters
> 2015-04-17 13:46:22,678 DEBUG [c.c.d.FirstFitPlanner]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) No clusters
> found after removing disabled clusters and clusters in avoid list,
> returning.
> 2015-04-17 13:46:22,683 DEBUG [c.c.v.UserVmManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Destroying vm
> VM[User|i-15-1151-VM] as it failed to create on Host with Id:null
> 2015-04-17 13:46:22,706 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) VM state
> transitted from :Stopped to Error with event:
> OperationFailedToErrorvm's original host id: null new host id: null
> host id before state transition: null
> 2015-04-17 13:46:22,724 WARN  [o.a.c.alerts]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb)  alertType::
> 8 // dataCenterId:: 1 // podId:: null // clusterId:: null // message::
> Failed to deploy Vm with Id: 1151, on Host with Id: null
> 2015-04-17 13:46:22,780 INFO  [o.a.c.a.c.u.v.DeployVMCmd]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb)
> com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a deployment for VM[User|i-15-1151-VM]Scope=interface
> com.cloud.dc.DataCenter; id=1
> 2015-04-17 13:46:22,781 INFO  [o.a.c.a.c.u.v.DeployVMCmd]
> (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Unable to
> create a deployment for VM[User|i-15-1151-VM]
> com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a deployment for VM[User|i-15-1151-VM]Scope=interface
> com.cloud.dc.DataCenter; id=1
> at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:214)
> at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:200)
> at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3468)
> at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3123)
> at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy207.startVirtualMachine(Unknown Source)
> at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:449)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-17 13:46:22,783 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-16:ctx-e039eccb job-7312) Complete async job-7312,
> jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":533,"errortext":"Unable
> to create a deployment for VM[User|i-15-1151-VM]"}
> 2015-04-17 13:46:22,794 DEBUG [o.a.c.f.j.i.AsyncJobManage



Mime
View raw message