cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Somesh Naidu <Somesh.Na...@citrix.com>
Subject RE: XenServer is disconnected after CS hosts shutdown
Date Wed, 12 Aug 2015 15:02:09 GMT
> starting XS agent manually

Either unmanage/manage cluster or perform a force reconnect to the host.

Regards,
Somesh


-----Original Message-----
From: tony_caotong@163.com [mailto:tony_caotong@163.com] 
Sent: Wednesday, August 12, 2015 5:14 AM
To: users@cloudstack.apache.org
Subject: Re: XenServer is disconnected after CS hosts shutdown


After I disabled and enabled XS cluster & primary. it works.

seems XS agent have started by ACS host.   so what is the correct 
behavior of starting XS agent manually ?

-----------
Cao Tong

On 08/12/2015 04:41 PM, tony_caotong@163.com wrote:
>
> Hello,
>
> Almost one month gone, my problem is still here. I really really need 
> someone to help me.
>
> new Settings  ACS 4.4.4  XS 6.2
> After reboot, errors here:
>
> 2015-08-12 16:18:54,557 INFO  [c.c.a.t.Request] 
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]', 
> _cmds.length == 1
> 2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache] 
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding 
> null to 191386435611186
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache] 
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from 
> 249082151178140
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache] 
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId 
> 249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link 
> is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req: 
> Routing to peer
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req: 
> Cancel request received
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] 
> (AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] 
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting 
> some more time because this is the current command
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] 
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting 
> some more time because this is the current command
> 2015-08-12 16:18:54,559 INFO  [c.c.u.e.CSExceptionErrorCode] 
> (StatsCollector-3:ctx-b287085a) Could not find exception: 
> com.cloud.exception.OperationTimedoutException in error code list for 
> exceptions
> 2015-08-12 16:18:54,559 INFO  [c.c.a.t.Request] 
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]', 
> _cmds.length == 1
> 2015-08-12 16:18:54,559 WARN  [c.c.a.m.AgentAttache] 
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out 
> on null
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] 
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl] 
> (StatsCollector-3:ctx-b287085a) Unable to send storage pool command to 
> Pool[3|NetworkFilesystem] via 9
> com.cloud.exception.OperationTimedoutException: Commands 
> 4450963806725603425 to Host 9 timed out after 3600
>         at 
> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
>         at 
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
>         at 
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>         at 
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
>         at 
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
>         at 
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
>         at 
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>         at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>>
>> Hi Yiping,  Thanks for your reply first.
>>
>>
>> My NFS server deployed in ACS manager host,  another host is a single 
>> xenserver.
>> A KVM environment deployed in ACS manager host too.
>> there is three storages named, Kprimary, Xprimary, Secondary.
>>
>> First, I add KVM cluster with zone-wide Kprimary, it works fine even 
>> if after reboot.
>> Second,  I add Xenserver with Xprimary(cluster-wide),  it still works 
>> fine.
>> Then,  I set zone to disable,  shutdown all system VMs,  disable 
>> those two hosts.  finally shutdown xenserver host.
>> When xenserver's shutdown finish,  I stop those services by order ( 
>> cloudstack-agent,  cloudstack-management, libvirtd, nfs, rpcbind, 
>> mysqld).
>> At last shutdown ACS host.
>>
>> The start process is totally reverse of this order.
>>
>> when done of starting,  ACS says unable to send cmd to pool via host 
>> xenserver.
>>
>> I am sure nothing have change in my environment during reboot time.
>> When I fix this problem, the only change is change Kprimary from 
>> zone-wide to cluster-wide.
>>
>> I guess that after reboot some status have been initial from 
>> beginning,  ACS found that Xenserver host have two primary can be used.
>> so it use the high priority one, and it is Kprimary.
>>
>> Whatever, Maybe it could help you peoples to get ACS better.
>>
>> BTW, Error logs attached some days before is already point out this 
>> is a storage problem
>> like:
>>
>> Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>>
>>
>>
>> -----------
>> Cao Tong
>>
>> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>>> Well,  sometimes people can’t answer a question because of lack of
>>> relevant information, or simply because no one has encountered a 
>>> similar
>>> situation before.
>>>
>>> Looking at your past messages on this thread, there were no mentions 
>>> about
>>> primary storage. Obviously, your primary storage configuration had 
>>> changed
>>> between the time you shut down CS manager and xenservers and the 
>>> time you
>>> restarted them. That is the vital info the list didn’t know.
>>>
>>> To best of my knowledge, zone wide primary storage has never been
>>> supported for Xen hypervisors.
>>>
>>> I do have to say that quite often CloudStack error messages are very
>>> cryptic, do not provide enough *useful* information to help users 
>>> identify
>>> and trouble shoot actual problems. Those stack trace output might be a
>>> gold mine to developers, but they are utterly useless for end users.
>>>
>>> Just my $0.02
>>>
>>> Yiping
>>>
>>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <tony_caotong@163.com> 
>>> wrote:
>>>
>>>> Hi, Finally I resolved this problem by my self.
>>>>
>>>>   * Primary Storage: A storage resource typically provided to a single
>>>>     cluster for the actual running of instance disk images. (Zone-wide
>>>>     primary storage is an option, though not typically used.)
>>>>
>>>> This line above is from
>>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>>
>>>> Because I have a Zone-wide primary storage, ACS can not find the 
>>>> correct
>>>> primary which belong to XenServer cluster after reboot.
>>>>
>>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>>
>>>> Right now, I have two primary storage, one is kvm cluster-wide, 
>>>> another
>>>> is xenserver cluster-wide.
>>>>
>>>> Above is for people who have the same problem oneday.
>>>>
>>>> by the way, I am very curious why I never receive replys from this 
>>>> a big
>>>> community ??   of course except the very beginning.
>>>>
>>>> Is my English skill really really poor, result in no body can 
>>>> understood
>>>> what language I am speaking ?
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>>> Hey!  help please...
>>>>>
>>>>> some news.
>>>>> I think the cause is that the ACS host can't communicate with
>>>>> XenServer host.
>>>>> ACS continues outputing logs like this
>>>>>
>>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>>> Seq 5-8174877748607582212:  { Cmd , MgmtId: 279278805451459, via: 5,
>>>>> Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 280345368052992
>>>>>
>>>>> I am not sure that if the ACS status is wrong or some services on
>>>>> xenserver are not opend.
>>>>>
>>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>>
>>>>> As some operations about xenserver was pending, xenserver can not be
>>>>> deleted from web UI.
>>>>>
>>>>> I got a temporary solution
>>>>>
>>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>>> 2. delete xenserver from DB cloud.host.
>>>>> 3. add xenserver host back from web UI.
>>>>>
>>>>> then it works.
>>>>>
>>>>> Does anyone have a idea for this?
>>>>>
>>>>> Could anyone tell what things does ACS do on xenserver host when
>>>>> adding a xenserver ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>>> @prashant, following it the answer of you questions
>>>>>>
>>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>>
>>>>>> 2. No, Xenserver's password is not changed.
>>>>>>
>>>>>> 3. yes, web UI is fine, and I can login.
>>>>>>
>>>>>> 4.  before reboot, I unmanaged and disabled resources, and after
>>>>>> reboot I have enabled all of them.
>>>>>>
>>>>>> 5.  hosts is states is UP.
>>>>>>
>>>>>> 6. No yum update in anywhere.
>>>>>>
>>>>>> 7.  system VMs status is fine, i think.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> After reinstall, I got the problem again
>>>>>>>
>>>>>>> So, I will describe once again.
>>>>>>>
>>>>>>> WHAT my environment looks like:
>>>>>>>
>>>>>>> I have a ACS server host and a xenserver host, After both reboot,
I
>>>>>>> can not create a VM on xenserver through ACS.
>>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>>
>>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>>> create new VM using xencenter.
>>>>>>>
>>>>>>> ------------- ERR LOGS ----------
>>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>>
>>>>>>> 2015-07-22 15:56:56,358 INFO  [c.c.s.StatsCollector]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>>> Pool[4|NetworkFilesystem]
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:4] is unreachable: Unable to send command to the
pool
>>>>>>>
>>>>>>>
>>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>>> again and again -----------
>>>>>>>
>>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: 
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715922:  { Cmd , MgmtId: 227448510156211,

>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 116784073679673
>>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: 
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715883:  { Cmd , MgmtId: 227448510156211,

>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>>
>>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap

>>>>>>>
>>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5

>>>>>>>
>>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c

>>>>>>>
>>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30

>>>>>>>
>>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu

>>>>>>>
>>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS

>>>>>>>
>>>>>>>
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"

>>>>>>>
>>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-

>>>>>>>
>>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds

>>>>>>>
>>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com

>>>>>>>
>>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-

>>>>>>>
>>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c

>>>>>>>
>>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar

>>>>>>>
>>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po

>>>>>>>
>>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr

>>>>>>>
>>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr

>>>>>>>
>>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"

>>>>>>>
>>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe

>>>>>>>
>>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>>> } to 116784073679673
>>>>>>>
>>>>>>>
>>>>>>> -----------------------------------------
>>>>>>>
>>>>>>> Anyone have Any ideas?  thanks.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>>> Thanks all,
>>>>>>>>
>>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>>> environment to restart my research.
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>>> some questions :
>>>>>>>>>
>>>>>>>>> can you please tell ...
>>>>>>>>>
>>>>>>>>> 1. is your NFS storage or your primary Storage Repository
in
>>>>>>>>> connected
>>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>>> 3. is the cloudstack web ui up , can you login to the
cloudstack
>>>>>>>>> web page.
>>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled

>>>>>>>>> state
>>>>>>>>> ? *
>>>>>>>>> *5. is all the hosts in connected state  ? *
>>>>>>>>> *6. did you run  yum update on host reboot on the cs
manager 
>>>>>>>>> vm ? *
>>>>>>>>> *7. system vms are stateless you can kill them and cs
will
>>>>>>>>> recreate a new
>>>>>>>>> one .. so dont worry :-) *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *thanks *
>>>>>>>>> *prashant *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <tony_caotong@163.com>
wrote:
>>>>>>>>>
>>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver)
again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Following is the error log.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>>> com.cloud.exception.OperationTimedoutException in
error code 
>>>>>>>>>> list
>>>>>>>>>> for
>>>>>>>>>> exceptions
>>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
Timed
>>>>>>>>>> out on null
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>>> Cancelling.
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage
pool
>>>>>>>>>> command to
>>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>>>           at
>>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:

>>>>>>>>>>
>>>>>>>>>> 433)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:

>>>>>>>>>>
>>>>>>>>>> 362)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j

>>>>>>>>>>
>>>>>>>>>> ava:1000)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j

>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j

>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC

>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M

>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c

>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal

>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run

>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man

>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4

>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask

>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask

>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor

>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>>           at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>>> com.cloud.exception.StorageUnavailableException:
Resource
>>>>>>>>>> [StoragePool:1]
>>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j

>>>>>>>>>>
>>>>>>>>>> ava:1010)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j

>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j

>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC

>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M

>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c

>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal

>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run

>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man

>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4

>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask

>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask

>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor

>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>>           at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto

>>>>>>>>>>
>>>>>>>>>> r.java:615)
>>>>>>>>>>
>>>>>>>>>>           at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>>
>>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>>
>>>>>>>>>>> 1. In xenserver I can not login systemvms using
the internal IP
>>>>>>>>>>> like
>>>>>>>>>>> '169.254.1.112',  There shoud be a bridge network
for this
>>>>>>>>>>> right?  it is
>>>>>>>>>>> gone.
>>>>>>>>>>>
>>>>>>>>>>> 2. I try to delete xenserver host from CS on
web, it also 
>>>>>>>>>>> failed
>>>>>>>>>>> with
>>>>>>>>>>> lots of logs like following, then memory is full
and mangement
>>>>>>>>>>> down...
>>>>>>>>>>>
>>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399:  { Cmd , MgmtId: 104062526015411,

>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399:  { Cmd , MgmtId: 104062526015411,

>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399:  { Cmd , MgmtId: 104062526015411,

>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399:  { Cmd , MgmtId: 104062526015411,

>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess that,  is there some service or daemons
working for CS
>>>>>>>>>>> is not up
>>>>>>>>>>> on Xenserver ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----------
>>>>>>>>>>> Cao Tong
>>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>>> management server ip in the configuration
table is used my
>>>>>>>>>>>> systemvms.
>>>>>>>>>>>> select * from configuration where name like
'host';
>>>>>>>>>>>>
>>>>>>>>>>>> If it changed, correct the value in db and
restart systemvms.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ~Rajani
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<tony_caotong@163.com>
 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>    Hello,
>>>>>>>>>>>>> I shutdown my cs-manager and xenserver
last weekend, And now
>>>>>>>>>>>>> the ssvm
>>>>>>>>>>>>> and cpvm is disconnect, thost two was
runing on xenserver. so
>>>>>>>>>>>>> What
>>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In xenserver  I found that the three
system VMs are not 
>>>>>>>>>>>>> running.
>>>>>>>>>>>>> my xenserver seems can not reconnect
to CS-manager. and it
>>>>>>>>>>>>> seams not
>>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the right steps of shutdown all
CS group machines and
>>>>>>>>>>>>> resume
>>>>>>>>>>>>> them?
>>>>>>>>>>>>> How can i let my xenserver reconnected
?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> -----------
>>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
>>
>
>


Mime
View raw message