Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4857718FB7 for ; Wed, 26 Aug 2015 23:21:46 +0000 (UTC) Received: (qmail 40446 invoked by uid 500); 26 Aug 2015 23:21:46 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 40401 invoked by uid 500); 26 Aug 2015 23:21:46 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 40392 invoked by uid 500); 26 Aug 2015 23:21:46 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 40388 invoked by uid 99); 26 Aug 2015 23:21:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2015 23:21:46 +0000 Date: Wed, 26 Aug 2015 23:21:46 +0000 (UTC) From: "Remi Bergsma (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-8713) HA is not working on CentOS 7 due to KVM Power state report not properly parsed (Exception) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-8713?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D14715727#comment-14715727 ]=20 Remi Bergsma commented on CLOUDSTACK-8713: ------------------------------------------ Not sure how I got into this state. If I deploy a new cloud based on latest= master, I'm also not able to reproduce. > HA is not working on CentOS 7 due to KVM Power state report not properly = parsed (Exception) > -------------------------------------------------------------------------= ------------------ > > Key: CLOUDSTACK-8713 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-871= 3 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Affects Versions: 4.6.0 > Environment: KVM on CentOS 7, management server running latest ma= ster aka 4.6.0 > Reporter: Remi Bergsma > Priority: Critical > > While testing a PR, I found that HA on KVM does not work properly.=20 > Steps to reproduce: > - Spin up some VMs on KVM using a HA offering > - go to KVM hypervisor and kill one of them to simulate a crash > virsh destroy 6 (change number) > - look how cloudstack handles this missing VM > Result: > - VM stays down and is not started > Expected result: > - VM should be started somewhere > Cause: > It doesn=E2=80=99t parse the power report property it gets from the hyper= visor, so it never marks it Stopped. HA will not start, VM will stay down. > Database reports PowerStateMissing. Starting manually works fine. > select name,power_state,instance_name,state from vm_instance where name= =3D'test003'; > | name | power_state | instance_name | state | > | test003 | PowerReportMissing | i-2-6-VM | Running | > 1 row in set (0.00 sec) > I also tried to crash a KVM hypervisor and then the same thing happens. > Haven=E2=80=99t tested it on other hypervisors. Could anyone verify this? > Logs: > 2015-08-06 15:40:46,809 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (A= gentManager-Handler-16:null) VM state report is updated. host: 1, vm id: 6,= power state: PowerReportMissing=20 > 2015-08-06 15:40:46,815 INFO [c.c.v.VirtualMachineManagerImpl] (AgentMan= ager-Handler-16:null) VM i-2-6-VM is at Running and we received a power-off= report while there is no pending jobs on it > 2015-08-06 15:40:46,815 INFO [c.c.v.VirtualMachineManagerImpl] (AgentMan= ager-Handler-16:null) Detected out-of-band stop of a HA enabled VM i-2-6-VM= , will schedule restart > 2015-08-06 15:40:46,824 INFO [c.c.h.HighAvailabilityManagerImpl] (AgentM= anager-Handler-16:null) Schedule vm for HA: VM[User|i-2-6-VM] > 2015-08-06 15:40:46,824 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (A= gentManager-Handler-16:null) Done with process of VM state report. host: 1 > 2015-08-06 15:40:46,851 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Wor= ker-3:ctx-4e073b92 work-37) Processing HAWork[37-HA-6-Running-Investigating= ] > 2015-08-06 15:40:46,871 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Wor= ker-3:ctx-4e073b92 work-37) HA on VM[User|i-2-6-VM] > 2015-08-06 15:40:46,880 DEBUG [c.c.a.t.Request] (HA-Worker-3:ctx-4e073b92= work-37) Seq 1-6463228415230083145: Sending { Cmd , MgmtId: 3232241215, v= ia: 1(kvm2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckVirtualMac= hineCommand":{"vmName":"i-2-6-VM","wait":20}}] } > 2015-08-06 15:40:46,908 ERROR [c.c.a.t.Request] (AgentManager-Handler-17:= null) Unable to convert to json: [{"com.cloud.agent.api.CheckVirtualMachine= Answer":{"state":"Stopped","result":true,"contextMap":{},"wait":0}}] > 2015-08-06 15:40:46,909 WARN [c.c.u.n.Task] (AgentManager-Handler-17:nul= l) Caught the following exception but pushing on > com.google.gson.JsonParseException: The JsonDeserializer EnumTypeAdapter = failed to deserialize json object "Stopped" given the type class com.cloud.= vm.VirtualMachine$PowerState > at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(J= sonDeserializerExceptionWrapper.java:64) > at com.google.gson.JsonDeserializationVisitor.invokeCustomDeseria= lizer(JsonDeserializationVisitor.java:92) > at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsi= ngCustomHandler(JsonObjectDeserializationVisitor.java:117) > at com.google.gson.ReflectingFieldNavigator.visitFieldsReflective= ly(ReflectingFieldNavigator.java:63) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:12= 0) > at com.google.gson.JsonDeserializationContextDefault.fromJsonObje= ct(JsonDeserializationContextDefault.java:76) > at com.google.gson.JsonDeserializationContextDefault.deserialize(= JsonDeserializationContextDefault.java:54) > at com.google.gson.Gson.fromJson(Gson.java:551) > at com.google.gson.Gson.fromJson(Gson.java:521) > at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTy= peAdaptor.java:80) > at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTy= peAdaptor.java:40) > at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(J= sonDeserializerExceptionWrapper.java:51) > at com.google.gson.JsonDeserializationVisitor.invokeCustomDeseria= lizer(JsonDeserializationVisitor.java:92) > at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHan= dler(JsonDeserializationVisitor.java:80) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:10= 1) > at com.google.gson.JsonDeserializationContextDefault.fromJsonArra= y(JsonDeserializationContextDefault.java:67) > at com.google.gson.JsonDeserializationContextDefault.deserialize(= JsonDeserializationContextDefault.java:52) > at com.google.gson.Gson.fromJson(Gson.java:551) > at com.google.gson.Gson.fromJson(Gson.java:498) > at com.google.gson.Gson.fromJson(Gson.java:467) > at com.google.gson.Gson.fromJson(Gson.java:417) > at com.google.gson.Gson.fromJson(Gson.java:389) > at com.cloud.agent.transport.Request.log(Request.java:399) > at com.cloud.agent.transport.Request.logD(Request.java:368) > at com.cloud.agent.manager.AgentAttache.processAnswers(AgentAttac= he.java:271) > at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAge= ntHandler.doTask(ClusteredAgentManagerImpl.java:709) > at com.cloud.utils.nio.Task.run(Task.java:84) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: No enum constant com.cloud= .vm.VirtualMachine.PowerState.Stopped > at java.lang.Enum.valueOf(Enum.java:236) > at com.google.gson.DefaultTypeAdapters$EnumTypeAdapter.deserializ= e(DefaultTypeAdapters.java:524) > at com.google.gson.DefaultTypeAdapters$EnumTypeAdapter.deserializ= e(DefaultTypeAdapters.java:514) > at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(J= sonDeserializerExceptionWrapper.java:51) > ... 29 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)