cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tilak Raj Singh <tila...@gmail.com>
Subject Re: Unable to spawn new System VMs after server restarted
Date Thu, 16 Jul 2015 19:55:46 GMT
Hi Somesh,

Sorry for late reply. I ctually deleted all the entries from
op_dc_ip_address alloc table as the cpvm was hung in start phase...the
issue started after that only..I then again entered a dummy entry with
corredponding reservation id and nic id uing nic table and vm_instance
table but still the error is same. Can you guide me which files or db dump
to attach so that you can help me resolve the issue

Regards

Tilak Raj Singh

On Fri, Jul 10, 2015 at 9:51 PM, Somesh Naidu <Somesh.Naidu@citrix.com>
wrote:

> Need more logs (the relevant exception stack to be precise) to locate the
> root cause of the error.
>
> In addition, can you check the integrity of the following table/view (just
> perform a select on these):
> async_job_view
> op_dc_ip_address_alloc
>
> Regards,
> Somesh
>
>
> -----Original Message-----
> From: Tilak Raj Singh [mailto:tilak72@gmail.com]
> Sent: Thursday, July 09, 2015 1:36 PM
> To: users@cloudstack.apache.org
> Subject: Re: Unable to spawn new System VMs after server restarted
>
> hi all...can somebody please guide me through this process
>
> Regards
>
> On Wed, Jul 8, 2015 at 8:25 PM, Tilak Raj Singh <tilak72@gmail.com> wrote:
>
> > Hi All,
> >
> > Recently my Server started from backup and last I checked all my
> > Management IPs were full and hence Cloudstack was not able to spawn new
> > System Vms.
> > Trying to solv the issue i cleared all the allocated management ips from
> > op_dc_ip_address_alloc table in database. But now I am getting this issue
> > of Foreign Key constraint when the management server is trying to
> expunge a
> > whose entry was there in the database. As a consequence new system vms
> are
> > not able to spawn up. Can somebody help me with details as to which
> entries
> > and in which tables should I delete for all this to start working again
> or
> > any other help in this regard.
> >
> > Here is the error log
> >
> > 2015-07-08 20:16:54,880 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Expunging unfinished job
> AsyncJobVO
> > {id:15864, userId: 1, accountId: 1, instanceType: null, instanceId: null,
> > cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAJMHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 91019618482122, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: Tue Jun 30 00:59:18 IST 2015}
> > 2015-07-08 20:16:54,882 DEBUG [c.c.u.d.T.Transaction]
> > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Rolling back the transaction:
> Time =
> > 2 Name =  AsyncJobMgr-Heartbeat-1; called by
> >
> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-TransactionLegacy.close:667-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy226.expunge:-1-AsyncJobManagerImpl$8.doInTransactionWithoutResult:813-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49
> > 2015-07-08 20:16:54,908 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Unexpected exception when trying
> to
> > execute queue item,
> > com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> > com.mysql.jdbc.PreparedStatement@1ea2e3c8: DELETE FROM async_job WHERE
> > async_job.id= 15864
> > at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
> > at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> > at
> >
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> > at com.sun.proxy.$Proxy226.expunge(Unknown Source)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$8.doInTransactionWithoutResult(AsyncJobManagerImpl.java:813)
> > at
> >
> com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
> > at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.expungeAsyncJob(AsyncJobManagerImpl.java:810)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.reallyRun(AsyncJobManagerImpl.java:773)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.runInContext(AsyncJobManagerImpl.java:749)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by:
> >
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> > Cannot delete or update a parent row: a foreign key constraint fails
> > (`cloud`.`async_job_join_map`, CONSTRAINT
> > `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`)
> REFERENCES
> > `async_job` (`id`))
> > at sun.reflect.GeneratedConstructorAccessor116.newInstance(Unknown
> Source)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at com.mysql.jdbc.Util.getInstance(Util.java:386)
> > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
> > ... 32 more
> >
> > Regards
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message