Mailing-List: contact cloudstack-users-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-users@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of kirkkosinski@gmail.com
 designates 209.85.160.47 as permitted sender)
Message-ID: <50CE64F8.6090405@gmail.com>
Date: Sun, 16 Dec 2012 16:19:04 -0800
From: Kirk Kosinski <kirkkosinski@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:10.0.11) Gecko/20121203 Thunderbird/10.0.11
MIME-Version: 1.0
To: cloudstack-users@incubator.apache.org
Subject: Re: Damn time travel
References: 
 <CACb7MQVdw+4+GBGAAEPTx=uSJfKjynwcgAOd1cFRgrn8ncshhA@mail.gmail.com>
In-Reply-To: 
 <CACb7MQVdw+4+GBGAAEPTx=uSJfKjynwcgAOd1cFRgrn8ncshhA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi, Dave.  What does the VM and it's volumes look like in the database?

select * from vm_instance where id = 620;
select * from volumes where instance_id = 620;

Best regards,
Kirk


On 12/16/2012 03:56 PM, David Comerford wrote:
> Hey,
> 
> Due to some clocks being a few months forward for a while I'm stuck back in
> the present with a number of VMs that just won't expunge.
> Logs suggest there's some concurrent operations on said VM which I think
> basically means theres some job that lives in the future which is blocking
> the current ones.
> 
> My questions:
> 1) From the logs below, does my guess sound plausible?
> 2) Where in the database is the queue for jobs I need to take a hammer to?
> 
> Cheers guys,
> Dave
> 
> 2012-12-16 23:44:01,132 WARN  [cloud.vm.UserVmManagerImpl]
> (AccountChecker-1:null) Concurrent operations on expunging
> VM[User|a1a470ef-368b-45fe-99c7-1b92ea333d78]
> com.cloud.exception.ConcurrentOperationException: Failed to transit state
>         at
> com.cloud.storage.StorageManagerImpl.destroyVolume(StorageManagerImpl.java:2060)
>         at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30)
>         at
> com.cloud.storage.StorageManagerImpl.cleanupVolumes(StorageManagerImpl.java:3486)
>         at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30)
>         at
> com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:391)
>         at
> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1160)
>         at
> com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:518)
>         at
> com.cloud.user.AccountManagerImpl$AccountCleanupTask.run(AccountManagerImpl.java:1291)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2012-12-16 23:44:01,132 ERROR [cloud.user.AccountManagerImpl]
> (AccountChecker-1:null) Unable to expunge vm: 620
>