Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: domain of refond.gmrt@gmail.com
 designates 74.125.82.179 as permitted sender)
MIME-Version: 1.0
Date: Mon, 5 Jan 2015 19:55:40 +0530
Message-ID: 
 <CAGB-CvTR5uYDf6p=HKL8EoNwmi1u+MZ7D289xC4VBV6wS8re1Q@mail.gmail.com>
Subject: Unable to add primary storage (SR_BACKEND_FAILURE_73NFS mount error
 [opterr=mount failed with return code 32])
From: Tejas Gadaria <refond.gmrt@gmail.com>
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>,
	"dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary=047d7b5d348c079fe6050be877ee

--047d7b5d348c079fe6050be877ee
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

I am facing issue with xenserver 6.2 SP1, on ACS 4.3.1 and 4.4 both.


While creating Zone with NFS as primary & secondary storage

2015-01-05 19:29:45,715 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Executing request
2015-01-05 19:29:47,079 WARN  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-2:ctx-23cfb66b) Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
com.cloud.utils.exception.CloudRuntimeException: Unable to create NFS SR
Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary]
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourc=
eBase.java:6133)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResource=
Base.java:5074)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixR=
esourceBase.java:475)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenSer=
ver56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenSe=
rver610Resource.java:102)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtt=
ache.java:307)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedC=
ontextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Def=
aultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCo=
ntext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCon=
text(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCon=
textRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access=
$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc=
heduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1=
145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:=
615)
at java.lang.Thread.run(Thread.java:745)
Caused by: SR_BACKEND_FAILURE_73NFS mount error [opterr=3Dmount failed with
return code 32]
at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnecti=
on.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.SR.create(SR.java:801)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourc=
eBase.java:6129)
... 17 more
2015-01-05 19:29:47,080 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Response Received:
2015-01-05 19:29:47,080 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Response Received:
2015-01-05 19:29:47,080 DEBUG [c.c.a.t.Request]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Processing:  { Ans:
, MgmtId: 158445874166998, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary","wait":0}}] }
2015-01-05 19:29:47,081 DEBUG [c.c.a.t.Request]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Seq 1-3875347479352311816:
Received:  { Ans: , MgmtId: 158445874166998, via: 1, Ver: v1, Flags: 10, {
Answer } }
2015-01-05 19:29:47,081 DEBUG [c.c.a.m.AgentManagerImpl]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Details from executing class
com.cloud.agent.api.CreateStoragePoolCommand: Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
2015-01-05 19:29:47,126 WARN
 [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Can not create storage pool
through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
2015-01-05 19:29:47,126 DEBUG [c.c.s.StorageManagerImpl]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Failed to add data store: Can
not create storage pool through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
com.cloud.utils.exception.CloudRuntimeException: Can not create storage
pool through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStor=
eLifeCycleImpl.createStoragePool(CloudStackPrimaryDataStoreLifeCycleImpl.ja=
va:384)
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStor=
eLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:4=
07)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:669=
)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:178=
)
at
org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execut=
e(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:682)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:511)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:330)
at com.cloud.api.ApiServlet.access$000(ApiServlet.java:54)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:118)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Def=
aultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCo=
ntext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCon=
text(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:115)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio=
nFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC=
hain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j=
ava:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j=
ava:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12=
7)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10=
2)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555=
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav=
a:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java=
:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(=
Http11NioProtocol.java:721)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java=
:2274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1=
145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:=
615)
at java.lang.Thread.run(Thread.java:745)
2015-01-05 19:29:47,129 INFO  [c.c.a.ApiServer]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) null
2015-01-05 19:29:47,130 DEBUG [c.c.a.ApiServlet]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) =3D=3D=3DEND=3D=3D=3D  192.168=
.160.95 --
GET
 command=3DcreateStoragePool&zoneid=3D123cf76d-077a-4d92-a4e1-708aca573b1a&=
podId=3D5041c821-0d10-47df-a363-01f87cccfc00&clusterid=3D19698fe8-2921-48ea=
-9158-608cac69b5ac&name=3DPrimary&scope=3Dcluster&url=3Dnfs%3A%2F%2F192.168=
.161.110%2Fmnt%2Fdata%2FVMS-TT%2Fprimary&response=3Djson&sessionkey=3DqT%2F=
MwkWgbzKOyRSPltz6ETNUkeE%3D&_=3D1420466385645
2015-01-05 19:30:01,600 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-c4bb0888) Skip capacity scan due to there is no Primary
Storage UPintenance mode


Same time Xenserver host showing following log..


xenserver-01 xapi: [error|xenserver-01|926 INET 0.0.0.0:80|SR.create
R:adea9138f9b1|storage_access] Re-raising as SR_BACKEND_FAILURE_73 [ ; NFS
mount error [opterr=3Dmount failed with return code 32];  ]
Jan  5 19:29:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:c87d8d71bfdd|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:29:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:11d92819dc30|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:30:17 xenserver-01 last message repeated 2 times
Jan  5 19:30:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:d575526e384a|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:30:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:d575526e384a|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:30:43 xenserver-01 heartbeat: Problem with heartbeat, no iSCSI or
NFS mount defined in /opt/cloud/bin/heartbeat!
Jan  5 19:30:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:d575526e384a|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:30:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:6d4a45ec4259|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:31:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:6d4a45ec4259|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:31:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:0ecee91bf125|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:31:43 xenserver-01 heartbeat: Problem with heartbeat, no iSCSI or
NFS mount defined in /opt/cloud/bin/heartbeat!
Jan  5 19:31:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:0ecee91bf125|xapi] get_since_for_events: no in_memory_cache!
Jan  5 19:31:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:f473b74de5fd|xapi] get_since_for_events: no in_memory_cache!


Tried to stop iptables on XS host
I checked /opt/cloud/bin/heartbeat file, on XS host and file is empty.


Need help on this..


Regards,
Tejas

--047d7b5d348c079fe6050be877ee--