cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jochim, Ingo" <Ingo.Joc...@bautzen-it.de>
Subject AW: ACS does shutdown some VMs
Date Thu, 25 Jun 2015 06:14:44 GMT
Hello all,

any idea how to cleanup the job table?

Thanks and regards,
Ingo

-----Ursprüngliche Nachricht-----
Von: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de] 
Gesendet: Dienstag, 23. Juni 2015 14:59
An: 'users@cloudstack.apache.org'
Betreff: AW: ACS does shutdown some VMs

Hello Remi,

ACS has version 4.4.1.

This is the worker output. We never configured HA. How can we get rid of this?

+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM     |    Type   | VM state |    Created (-2H)    | HAworker step taken |    Step   |
Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running  | 2015-06-22 14:40:23 |         None        | Scheduled |
   hv01    |    None    |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.


The job queue is also quite long. What happens there?
Regards,
Ingo 


+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name |  vm_state |                               job_cmd
                               |     job_dispatcher    |     Job created     | Mgt Server
| Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
|  user01  |    admin     | None          |    None   |                      com.cloud.vm.VmWorkReboot
                      |  VmWorkJobDispatcher  | 2015-06-22 11:04:30 |    None    |  3791 
|      3790      |
|  user01  |    admin     | None          |    None   |                  com.cloud.vm.VmWorkRemoveNicFromVm
                 |  VmWorkJobDispatcher  | 2015-06-23 10:30:01 |    None    |  3860  |   
  3859      |
|  user01  |    admin     | None          |    None   |                       com.cloud.vm.VmWorkStart
                      |  VmWorkJobDispatcher  | 2015-06-22 12:26:21 |    None    |  3817 
|      3816      |
|  user01  |    admin     | None          |    None   |                       com.cloud.vm.VmWorkStart
                      |  VmWorkJobDispatcher  | 2015-06-22 13:38:06 |    None    |  3832 
|      3831      |
|  user01  |    admin     | None          |    None   |                       com.cloud.vm.VmWorkStart
                      |  VmWorkJobDispatcher  | 2015-06-23 06:55:58 |    None    |  3857 
|      3856      |
|  user01  |    admin     | None          |    None   |                       com.cloud.vm.VmWorkStop
                       |  VmWorkJobDispatcher  | 2015-06-22 09:21:34 |    None    |  3778
 |      3777      |
|  user01  |    admin     | None          |    None   |                       com.cloud.vm.VmWorkStop
                       |  VmWorkJobDispatcher  | 2015-06-22 10:55:04 |    None    |  3789
 |      3788      |
|  user01  |    admin     | None          |    None   | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin
| ApiAsyncJobDispatcher | 2015-06-23 10:30:01 |    None    |  3859  |                |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-18 08:01:43 |    None    |  3601  | 
    3590      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-18 08:01:45 |    None    |  3602  | 
    3567      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-18 08:01:47 |    None    |  3604  | 
    3559      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-18 08:03:07 |    None    |  3616  | 
    3612      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-20 11:53:36 |    None    |  3704  | 
    3609      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-20 12:34:21 |    None    |  3718  | 
    3610      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-20 12:34:21 |    None    |  3722  | 
    3611      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-20 12:34:49 |    None    |  3733  | 
    3609      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-20 13:01:36 |    None    |  3743  | 
    3610      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-20 13:01:44 |    None    |  3749  | 
    3606      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 12:08:26 |    None    |  3796  | 
    3698      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 12:08:27 |    None    |  3798  | 
    3607      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 12:09:32 |    None    |  3809  | 
    3615      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 13:37:15 |    None    |  3825  | 
    3819      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 14:40:24 |    None    |  3836  | 
    3818      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 14:40:24 |    None    |  3837  | 
    3556      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 14:40:24 |    None    |  3839  | 
    3559      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 14:40:24 |    None    |  3841  | 
    3570      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 14:40:24 |    None    |  3842  | 
    3567      |
|  system  |    system    | None          |    None   |                    com.cloud.vm.VmWorkMigrateAway
                   |  VmWorkJobDispatcher  | 2015-06-22 14:40:24 |    None    |  3845  | 
    3819      |
|  system  |    system    | None          |    None   |                       com.cloud.vm.VmWorkStart
                      |  VmWorkJobDispatcher  | 2015-06-18 07:59:09 |    None    |  3600 
|      3578      |
|  user01  |    admin     | i-116-616-VM  |  Stopped  |    org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin
    | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 |    None    |  3790  |                |
|  user01  |    admin     | i-116-616-VM  |  Stopped  |     org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
    | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 |    None    |  3816  |                |
|  user01  |    admin     | i-116-616-VM  |  Stopped  |     org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
    | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 |    None    |  3831  |                |
|  user01  |    admin     | i-116-616-VM  |  Stopped  |     org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
    | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 |    None    |  3835  |                |
|  user01  |    admin     | i-116-616-VM  |  Stopped  |     org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
    | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 |    None    |  3856  |                |
|  user01  |    admin     | i-116-616-VM  |  Stopped  |     org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin
     | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 |    None    |  3777  |               
|
|  user01  |    admin     | i-116-616-VM  |  Stopped  |     org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin
     | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 |    None    |  3788  |               
|
|  system  |    system    | s-32-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-20 11:53:33 |    None  
 |  3698  |                |
|  system  |    system    | s-33-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3609  |                |
|  system  |    system    | s-34-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3606  |                |
|  system  |    system    | s-35-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3611  |                |
|  system  |    system    | s-36-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3610  |                |
|  system  |    system    | s-37-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:03 |    None  
 |  3615  |                |
|  system  |    system    | s-38-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:03 |    None  
 |  3614  |                |
|  system  |    system    | s-39-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3612  |                |
|  system  |    system    | s-40-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3613  |                |
|  system  |    system    | s-41-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3607  |                |
|  system  |    system    | s-42-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:02 |    None  
 |  3608  |                |
|  system  |    system    | s-43-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:47 |    None  
 |  3617  |                |
|  system  |    system    | s-44-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 08:03:47 |    None  
 |  3618  |                |
|  system  |    system    | s-45-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-20 12:35:04 |    None  
 |  3734  |                |
|  system  |    system    | s-82-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:52:09 |    None  
 |  3578  |                |
|  system  |    system    | s-83-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-22 13:37:12 |    None  
 |  3819  |                |
|  system  |    system    | s-84-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-22 13:37:12 |    None  
 |  3818  |                |
|  system  |    system    | s-85-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:52:09 |    None  
 |  3579  |                |
|  system  |    system    | s-86-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:25 |    None  
 |  3559  |                |
|  system  |    system    | s-87-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:26 |    None  
 |  3570  |                |
|  system  |    system    | s-88-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:25 |    None  
 |  3558  |                |
|  system  |    system    | s-89-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:25 |    None  
 |  3560  |                |
|  system  |    system    | s-90-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:25 |    None  
 |  3556  |                |
|  system  |    system    | s-91-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:25 |    None  
 |  3557  |                |
|  system  |    system    | s-92-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:26 |    None  
 |  3567  |                |
|  system  |    system    | s-93-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:57:41 |    None  
 |  3590  |                |
|  system  |    system    | s-94-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:25 |    None  
 |  3561  |                |
|  system  |    system    | s-95-VM       | Expunging |                                 None
                                |  pseudoJobDispatcher  | 2015-06-18 07:51:26 |    None  
 |  3569  |                |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.


-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com] 
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs

Hi,

It’d double check the HA worker table to see if there are any left-over HA workers. Also,
there could still be other jobs scheduled. If the API does not return anything useful, you
may have a look at the DB itself.

I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well.
You can also execute the queries yourself, see [3] and [4] to get an idea.

Regards,
Remi

[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109

On 23 Jun 2015, at 07:27, Tomasz Zięba <t.a.zieba@gmail.com<mailto:t.a.zieba@gmail.com>>
wrote:

What version of ACS ?

Regards,
Tomasz Zięba

Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>

sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <Ingo.Jochim@bautzen-it.de<mailto:Ingo.Jochim@bautzen-it.de>>
napisał(a):

Hello Andrei,

The VM is not a HA VM.

How can I check the queue?

cloudmonkey list asyncjobs ???

isdefault = False # does this mean it should be off ?

Regards,
Ingo


________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<mailto:andrei@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs

Hi Ingo,

It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job.
Check out the jobs in the queue.

Andrei



----- Original Message -----

From: "Ingo Jochim" <Ingo.Jochim@bautzen-it.de<mailto:Ingo.Jochim@bautzen-it.de>>
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs

Hello all,

we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay
running.
Is there some kind of destination state which might be wrong in the database?

Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com


--
This email was Virus checked by UTM 9. http://www.sophos.com


Mime
View raw message