cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cs user <acldstk...@gmail.com>
Subject Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
Date Thu, 01 Oct 2015 14:09:10 GMT
Apologies for the spam.... :-)

Looks like it is this problem.

https://issues.apache.org/jira/browse/CLOUDSTACK-8412

No idea why this didn't appear during my earlier googling.

However, the commit which fixes this appears to already be in 4.5.1 and
4.5.2:

user@ubuntu01:~/cloudstack/cloudstack$ git tag --contains
27b7e49b395d089ff45037faa51e73d979ceb62b
4.5.1
4.5.2

However we have two different environments installed, one on version 4.5.1
and one with 4.5.2, and are seeing the same issue on both....

Cheers






On Thu, Oct 1, 2015 at 2:58 PM, cs user <acldstkusr@gmail.com> wrote:

> Hello,
>
> Found a better error in the logs:
>
> ERROR [c.c.a.t.Request] (AgentManager-Handler-11:null) Caught problem with
> [{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":26631,"name":"i-2-26631-VM","bootloader":"PyGrub","type":"Us
> er","cpus":1,"minSpeed":1200,"maxSpeed":1200,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
> 5.6 (64-bit)","platformEmulator":"CentOS 5
> (64-bit)","bootArgs":"","enableHA":false,"
>
> limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:1;apic:true;pae:true;nx:t
>
> rue","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"b37e76e8-e19a-4100-bf1c-1ee8eba674f2","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"8f51
>
> e73f-9bfc-4a37-bf86-664067a2cf40","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b114d58e-a37a-6433-c0ff-07f3613a3c68","id":219,"poolType":"LVM","ho
> st":"192.168.97.2","path":"lvm","port":0,"url":"LVM://
> 192.168.97.2/lvm/?ROLE\u003dPrimary\u0026STOREUUID\u003db114d58e-a37a-6433-c0ff-07f3613a3c68
> <http://192.168.97.2/lvm/?ROLE%5Cu003dPrimary%5Cu0026STOREUUID%5Cu003db114d58e-a37a-6433-c0ff-07f3613a3c68>
> "}},"name":"ROOT-26631","size":21474836480,"path":"48
>
> fe494b-ee02-4f08-b7aa-975baecd7b3e","volumeId":119075,"vmName":"i-2-26631-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":119075,"deviceId":0,"cacheMode":"NONE","hypervisorType":"Xe
>
> nServer"}},"diskSeq":0,"path":"48fe494b-ee02-4f08-b7aa-975baecd7b3e","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"192.168.97.2","volumeSize":"21474836480"}},{"data":{
>
> "org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":2000,"defaultNic":true,"pxe
>
> Disable":false,"nicUuid":"aaef8c14-784c-403f-91fc-b6a3167f2595","uuid":"508d0384-5bf0-4434-8155-0b5bb1b33b43","ip":"192.168.8.112","netmask":"255.255.254.0","gateway":"192.168.9.254","mac":"06:68:80:
>
> 00:00:7f","dns1":"192.168.0.224","dns2":"192.168.0.224","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":true,"name":"CS-Guest-Public"}],"vcpuMaxLimit":
>
> 16},"volumeToFiler":{"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]":{"id":223,"uuid":"bc01ca2b-e6d8-a856-27b5-6f6b55a06b1d","host":"192.168.97.1","path":"lvm","port":0,"type":"L
> VM"}},"contextMap":{"job":"job-347068/job-347069"},"wait":0}}]
> com.google.gson.JsonParseException: Expecting object found:
> "Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
>         at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
>         at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
>         at
> com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
>         at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
>         at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
>         at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
>         at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
>         at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
>         at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
>         at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
>         at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
>         at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
>         at com.google.gson.Gson.fromJson(Gson.java:551)
>         at com.google.gson.Gson.fromJson(Gson.java:521)
>         at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
>         at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
>         at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
>         at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
>         at
> com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
>         at
> com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
>         at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
>         at com.google.gson.Gson.fromJson(Gson.java:551)
>         at com.google.gson.Gson.fromJson(Gson.java:498)
>         at com.cloud.agent.transport.Request.getCommands(Request.java:250)
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
>         at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
>         at com.cloud.utils.nio.Task.run(Task.java:84)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> WARN  [c.c.u.n.Task] (AgentManager-Handler-11:null) Caught the following
> exception but pushing on
> com.google.gson.JsonParseException: Expecting object found:
> "Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
>         at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
>         at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
>         at
> com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
>         at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
>         at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
>         at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
>         at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
>         at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
>         at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
>         at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
>         at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
>         at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
>         at com.google.gson.Gson.fromJson(Gson.java:551)
>         at com.google.gson.Gson.fromJson(Gson.java:521)
>         at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
>         at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
>         at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
>         at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
>         at
> com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
>         at
> com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
>         at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
>         at com.google.gson.Gson.fromJson(Gson.java:551)
>         at com.google.gson.Gson.fromJson(Gson.java:498)
>         at com.cloud.agent.transport.Request.getCommands(Request.java:250)
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
>         at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
>         at com.cloud.utils.nio.Task.run(Task.java:84)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
>
>
>
> Not sure if that is more helpful?
>
> Cheers!
>
>
> On Thu, Oct 1, 2015 at 10:02 AM, cs user <acldstkusr@gmail.com> wrote:
>
>> Hi Koushik,
>>
>> We've managed to remove the error now regarding host id 27708.... this
>> was related to a very old secondary storage vm which was in a status of
>> disconnected but had not been removed. It had long since been deleted.
>>
>> However, we are still seeing the error, it is very similar to what is
>> described here:
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-3715
>>
>> Still seeing this:
>>
>> ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-6:ctx-8e19de38 job-346708/job-346709 ctx-65bccb97) Error
>> while migrating vm VM[User|i-2-26631-VM] to host Host[-181-Routing]
>> com.cloud.exception.OperationTimedoutException: Commands
>> 964614745187418300 to Host 181 timed out after 1200
>>
>> The migration never happens, its as though the call to the xen host is
>> never made, or never received by xen.
>>
>> What ports are used to make this call? I assume either 22 or 443?
>>
>> Cheers!
>>
>>
>>
>> On Wed, Sep 30, 2015 at 12:50 PM, Koushik Das <koushik.das@citrix.com>
>> wrote:
>>
>>> As per the logs the host id 27708 is not in the right state and
>>> operation is failing. Check in DB for the status of host id 27708.
>>> Also grep for job-343376 and job-343377 in all the MS logs to trace in
>>> which MS the job originated and where it is getting executed.
>>>
>>> -----Original Message-----
>>> From: cs user [mailto:acldstkusr@gmail.com]
>>> Sent: Wednesday, 30 September 2015 16:01
>>> To: users@cloudstack.apache.org
>>> Subject: Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>>>
>>> Hi Koushik,
>>>
>>> We see errors such as :
>>>
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>>> the right state: Disconnected ERROR
>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>> (Work-Job-Executor-159:ctx-6e40204e job-343376/job-343377 ctx-818f1a51)
>>> copy failed ERROR [c.c.v.VmWorkJobHandlerProxy]
>>> (Work-Job-Executor-159:ctx-6e40204e
>>> job-343376/job-343377 ctx-818f1a51) Invocation exception, caused by:
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>> VM[User|i-2-26383-VM] along with its volumes.
>>> INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-159:ctx-6e40204e
>>> job-343376/job-343377 ctx-818f1a51) Rethrow exception
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>> VM[User|i-2-26383-VM] along with its volumes.
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>> VM[User|i-2-26383-VM] along with its volumes.
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>>> the right state: Disconnected
>>>
>>> However, when I look at the hosts in cloudstack, all hosts are connected
>>> and green, none are reported by cloudstack as being disconnected. So I'm
>>> not really quite sure what is going wrong. As I say, when only one
>>> cloudstack management server is running, everything works perfectly fine.
>>>
>>> Cheers
>>>
>>> On Wed, Sep 30, 2015 at 10:27 AM, Koushik Das <koushik.das@citrix.com>
>>> wrote:
>>>
>>> > Check logs on all 4 MS in the cluster around the time the operation
>>> > was done. If you see any error/exceptions that can point to the
>>> problem.
>>> >
>>> > -----Original Message-----
>>> > From: cs user [mailto:acldstkusr@gmail.com]
>>> > Sent: Tuesday, 29 September 2015 20:53
>>> > To: users@cloudstack.apache.org
>>> > Subject: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>>> >
>>> > Hi Folks,
>>> >
>>> > We are seeing some problems when attempting live migrations in a
>>> > clustered cloudstack environment. We have 4 management servers, and
>>> > when we attempt a migration of an instance (local storage) from one
>>> > host to another, the migration times out.
>>> >
>>> > However, if we only have one management server running, the migration
>>> > works fine and we can move the instance around between hosts in the
>>> > same pod without any problems at all.
>>> >
>>> > Just wanted to know if anyone else had come across this issue?
>>> >
>>> > Cheers
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message