cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Xu <Xuefei...@citrix.com>
Subject RE: [4.4]copy_vhd_from_secondarystorage is timing out
Date Wed, 16 Apr 2014 18:27:14 GMT
Normally , SR.scan will not take too long, if SR.scan times out after 600 seconds, I think
there might be some problems with the SR,  I think you need to fix the SR issue first.
You can execute xe sr-scan in any XS hosts to see what's the problem.


Anthony

-----Original Message-----
From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com] 
Sent: Wednesday, April 16, 2014 11:21 AM
To: Anthony Xu
Cc: Murali Reddy; dev@cloudstack.apache.org
Subject: Re: [4.4]copy_vhd_from_secondarystorage is timing out

What I had done was change the xapiwait (or something like that) global setting and then re-start
my management server.


On Wed, Apr 16, 2014 at 12:19 PM, Mike Tutkowski < mike.tutkowski@solidfire.com> wrote:

> Ah, that's probably it. Thanks!
>
>
> On Wed, Apr 16, 2014 at 12:11 PM, Anthony Xu <Xuefei.Xu@citrix.com> wrote:
>
>>  I think you need to change value in DB configuration, changing the 
>> java code will not change the value in DB if the DB is already initialized.
>>
>>
>>
>>
>>
>>
>> +----------+----------+--------------+----------+-------+----------------------------------------------+---------------+---------+-------+------------+
>>
>> | Advanced | DEFAULT  | AgentManager | xapiwait | 60    | Time (in
>> seconds) to wait for XAPI to return | NULL          | NULL    | NULL
>> |          0 |
>>
>>
>>
>> *From:* Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
>> *Sent:* Wednesday, April 16, 2014 11:09 AM
>> *To:* Anthony Xu
>> *Cc:* Murali Reddy; dev@cloudstack.apache.org
>>
>> *Subject:* Re: [4.4]copy_vhd_from_secondarystorage is timing out
>>
>>
>>
>> Thanks, Anthony
>>
>>
>>
>> I will need to re-test. I changed that value from 60 to 600 last 
>> night and still saw an issue.
>>
>>
>>
>> Maybe I'll just start over from scratch and see if all is well.
>>
>>
>>
>> Thanks for the follow-up!
>>
>>
>>
>> On Wed, Apr 16, 2014 at 12:05 PM, Anthony Xu <Xuefei.Xu@citrix.com>
>> wrote:
>>
>> Just checked in a fix for this.
>>
>> change xapitimeout to 600 second,
>>
>> looks like we need to change some XAPI calls to Async before shorten 
>> the timeout
>>
>>
>>
>>
>>
>> Anthony
>>
>>
>>
>> *From:* Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
>> *Sent:* Wednesday, April 16, 2014 7:27 AM
>> *To:* Murali Reddy
>> *Cc:* Anthony Xu; dev@cloudstack.apache.org
>>
>>
>> *Subject:* Re: [4.4]copy_vhd_from_secondarystorage is timing out
>>
>>
>>
>> This is a similar problem to what I sent an e-mail out about last night:
>> XmlRpcException (Read timed out). For me it's happening when 
>> CitrixResourceBase issues SR.create.
>>
>>
>>
>> This issue is currently blocking my testing progress and I'm not 
>> really sure who might be the right person to look into this.
>>
>>
>>
>> Any thoughts on this anyone?
>>
>>
>>
>> On Wed, Apr 16, 2014 at 1:51 AM, Murali Reddy 
>> <Murali.Reddy@citrix.com>
>> wrote:
>>
>> System VM's are launching fine now after Anthony's fix. But user VM's 
>> are failing to get created with exception.
>>
>>
>>
>> 43275 2014-04-16 12:59:08,647 DEBUG 
>> [c.c.h.x.r.XenServerConnectionPool]
>> (DirectAgent-137:ctx-4761cb0c) XmlRpcException for method: 
>> VDI.snapshot due to org.apache.xmlrpc.XmlRpcException: Failed to 
>> create input stream: Read timed out
>>
>> 43276 2014-04-16 12:59:08,647 WARN 
>> [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-137:ctx-4761cb0c) Catch Exception 
>> org.apache.xmlrpc.XmlRpcException for template + due to
>> org.apache.xmlrpc.XmlRpcException: Failed to create input stream: 
>> Read timed out
>>
>>
>>
>> org.apache.xmlrpc.XmlRpcException: Failed to create input stream: 
>> Read timed out
>>
>> org.apache.xmlrpc.XmlRpcException: Failed to create input stream: 
>> Read timed out
>>
>> at
>> org.apache.xmlrpc.client.XmlRpcSunHttpTransport.getInputStream(XmlRpc
>> SunHttpTransport.java:99)
>>
>> at
>> org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStre
>> amTransport.java:152)
>>
>> at
>> org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTr
>> ansport.java:143)
>>
>> at
>> org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSun
>> HttpTransport.java:69)
>>
>> at
>> org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorke
>> r.java:56)
>>
>> at 
>> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
>>
>> at 
>> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)
>>
>> at 
>> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)
>>
>> at com.xensource.xenapi.Connection.dispatch(Connection.java:285)
>>
>> at
>> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerCo
>> nnection.dispatch(XenServerConnectionPool.java:456)
>>
>> at com.xensource.xenapi.VDI.snapshot(VDI.java:1155)
>>
>> at
>> com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTempl
>> ateToPrimaryStorage(XenServerStorageProcessor.java:972)
>>
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute
>> (StorageSubsystemCommandHandlerBase.java:77)
>>
>>
>>
>> *From: *Mike Tutkowski <mike.tutkowski@solidfire.com>
>> *Date: *Wednesday, 16 April 2014 12:35 AM
>> *To: *"dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
>> *Cc: *Murali Reddy <murali.reddy@citrix.com>
>>
>>
>> *Subject: *Re: [4.4]copy_vhd_from_secondarystorage is timing out
>>
>>
>>
>> Hi Anthony,
>>
>>
>>
>> Can you go into a bit more depth about copy_vhd_from_secondarystorage 
>> not being used with XenServer 6.2?
>>
>>
>>
>> Does that mean "public Answer 
>> copyTemplateToPrimaryStorage(CopyCommand
>> cmd)" in XenServerStorageProcessor is not called when XenServer 6.2 
>> is in use?
>>
>>
>>
>> I'm testing out a feature of mine for 4.4, but have only been using 
>> XenServer 6.1 so far.
>>
>>
>>
>> Thanks!
>>
>> Mike
>>
>>
>>
>> On Tue, Apr 15, 2014 at 12:44 PM, Anthony Xu <Xuefei.Xu@citrix.com>
>> wrote:
>>
>> Hi Murali,
>>
>> It is caused by the commit, I tested the commit on XS 6.2 FOX, which 
>> doesn't use copy_vhd_from_secondarystorage, that's why I didn't see 
>> this issue.
>>
>> Just disabled XAPI Event in 4.4 branch.
>>
>>
>> Thanks,
>> Anthony
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Murali Reddy
>> Sent: Tuesday, April 15, 2014 10:54 AM
>> To: dev@cloudstack.apache.org; Anthony Xu
>> Subject: Re: [4.4]copy_vhd_from_secondarystorage is timing out
>>
>>
>> I am still running into this issue. Alex reverted complete change 
>> replacing xapi.jar, which was also resulting XmlRpc exceptions but 
>> system VM's were launching fine after revert. I may be wrong but 
>> looking at the stack trace and recent commits, commit
>> 9f44909e63a19b2b8034baa13b92e24ea2eb0227 (use event instead of poll 
>> for xapi async call in XS 6.2 and above to reduce the pressure on 
>> XAPI) may be causing this issue. Anthony can you please take a look?
>>
>>
>> On 14/04/14 8:12 AM, "Mike Tutkowski" <mike.tutkowski@solidfire.com>
>> wrote:
>>
>> >I was seeing a similar issue when XAPI was changed when I tried to 
>> >create an SR.
>> >
>> >I wonder if that XAPI issue hasn't been fully resolved.
>> >
>> >
>> >On Sun, Apr 13, 2014 at 4:16 PM, Srikanteswararao Talluri < 
>> >srikanteswararao.talluri@citrix.com> wrote:
>> >
>> >> Did anybody see this on latest 4.4?
>> >>
>> >>
>> >> 2014-04-14 00:01:34,305 DEBUG [c.c.h.x.r.XenServerConnectionPool]
>> >> (DirectAgent-71:ctx-12d9be59) XmlRpcException for method: 
>> >>event.from due to
>> >> org.apache.xmlrpc.XmlRpcException: Failed to create input stream:
>> >>Read  timed out
>> >>
>> >> 2014-04-14 00:01:34,305 WARN  [c.c.h.x.r.CitrixResourceBase]
>> >> (DirectAgent-71:ctx-12d9be59) callHostPlugin failed for cmd:
>> >> copy_vhd_from_secondarystorage with args mountpoint:
>> >>10.147.28.7:/export/home/talluri/psec/template/tmpl/1/1/,
>> >> sruuid: b3dd21aa-8e82-a923-ed29-ee58220d158b, namelabel:
>> >> cloud-8498d88f-be6b-41b6-bece-a58bbc1fbc32,  due to Failed to 
>> >>create input
>> >> stream: Read timed out
>> >>
>> >> org.apache.xmlrpc.XmlRpcException: Failed to create input stream:
>> >> Read timed out
>> >>
>> >>         at
>> >>
>> >>org.apache.xmlrpc.client.XmlRpcSunHttpTransport.getInputStream(XmlR
>> >>pcS
>> >>unH
>> >>ttpTransport.java:99)
>> >>
>> >>         at
>> >>
>> >>org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcSt
>> >>rea
>> >>mTr
>> >>ansport.java:152)
>> >>
>> >>         at
>> >>
>> >>org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttp
>> >>Tra
>> >>nsp
>> >>ort.java:143)
>> >>
>> >>         at
>> >>
>> >>org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcS
>> >>unH
>> >>ttp
>> >>Transport.java:69)
>> >>
>> >>         at
>> >>
>> >>org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWor
>> >>ker
>> >>.ja
>> >>va:56)
>> >>
>> >>         at
>> >> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:16
>> >> 7)
>> >>
>> >>         at
>> >> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:13
>> >> 7)
>> >>
>> >>         at
>> >> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:12
>> >> 6)
>> >>
>> >>         at
>> >> com.xensource.xenapi.Connection.dispatch(Connection.java:285)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServer
>> >>Con
>> >>nec
>> >>tion.dispatch(XenServerConnectionPool.java:456)
>> >>
>> >>         at com.xensource.xenapi.Event.properFrom(Event.java:310)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.XenServer620Resource.waitForTask(
>> >>Xen
>> >>Ser
>> >>ver620Resource.java:125)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin
>> >>Asy
>> >>nc(
>> >>CitrixResourceBase.java:3558)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vh
>> >>d_f
>> >>rom
>> >>_secondarystorage(XenServerStorageProcessor.java:826)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTem
>> >>pla
>> >>teT
>> >>oPrimaryStorage(XenServerStorageProcessor.java:962)
>> >>
>> >>         at
>> >>
>> >>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execu
>> >>te(
>> >>Sto
>> >>rageSubsystemCommandHandlerBase.java:77)
>> >>
>> >>         at
>> >>
>> >>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handl
>> >>eSt
>> >>ora
>> >>geCommands(StorageSubsystemCommandHandlerBase.java:52)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest
>> >>(Ci
>> >>tri
>> >>xResourceBase.java:542)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.XenServer56Resource.executeReques
>> >>t(X
>> >>enS
>> >>erver56Resource.java:60)
>> >>
>> >>         at
>> >>
>> >>com.cloud.hypervisor.xen.resource.XenServer610Resource.executeReque
>> >>st(
>> >>Xen
>> >>Server610Resource.java:92)
>> >>
>> >>         at
>> >>
>> >>com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(Direct
>> >>Age
>> >>ntA
>> >>ttache.java:216)
>> >>
>> >>         at
>> >>
>> >>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(
>> >>Man
>> >>age
>> >>dContextRunnable.java:49)
>> >>
>> >>         at
>> >>
>> >>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.
>> >>cal
>> >>l(D
>> >>efaultManagedContext.java:56)
>> >>
>> >>         at
>> >>
>> >>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.ca
>> >>llW
>> >>ith
>> >>Context(DefaultManagedContext.java:103)
>> >>
>> >>
>> >>                                       547,2-9
>> >>
>> >
>> >
>> >
>> >--
>> >*Mike Tutkowski*
>> >*Senior CloudStack Developer, SolidFire Inc.*
>> >e: mike.tutkowski@solidfire.com
>> >o: 303.746.7302
>> >Advancing the way the world uses the 
>> >cloud<http://solidfire.com/solution/overview/?video=play>
>> >*(tm)*
>> >
>>
>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>>
>> *Senior CloudStack Developer, SolidFire Inc.*
>>
>> e: mike.tutkowski@solidfire.com
>>
>> o: 303.746.7302
>>
>> Advancing the way the world uses the 
>> cloud<http://solidfire.com/solution/overview/?video=play>
>> *(tm)*
>>
>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>>
>> *Senior CloudStack Developer, SolidFire Inc.*
>>
>> e: mike.tutkowski@solidfire.com
>>
>> o: 303.746.7302
>>
>> Advancing the way the world uses the 
>> cloud<http://solidfire.com/solution/overview/?video=play>
>> *(tm)*
>>
>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>>
>> *Senior CloudStack Developer, SolidFire Inc.*
>>
>> e: mike.tutkowski@solidfire.com
>>
>> o: 303.746.7302
>>
>> Advancing the way the world uses the 
>> cloud<http://solidfire.com/solution/overview/?video=play>
>> *(tm)*
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the 
> cloud<http://solidfire.com/solution/overview/?video=play>
> *(tm)*
>



--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*(tm)*

Mime
View raw message