incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkata Siva Vijayendra Bhamidipati (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-1228) Unable to Create System Vm's in the VMware Hypervisor setup.
Date Wed, 27 Feb 2013 01:27:11 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587842#comment-13587842
] 

Venkata Siva Vijayendra Bhamidipati commented on CLOUDSTACK-1228:
-----------------------------------------------------------------

Hi Kiran,

Anthony and I logged into 10.147.38.142 and took a look at the mgmt server process and the
directory structure.

[root@KiranRhel63-3 init.d]# service cloudstack-management status
cloudstack-management (pid  2941) is running...
[root@KiranRhel63-3 init.d]#



[root@KiranRhel63-3 init.d]# ps -ef | grep 2941
cloud     2941     1  3 Feb22 ?        03:59:27 /usr/lib/jvm/jre/bin/java -Djava.awt.headless=true
-Dcom.sun.management.jmxremote.port=45219 -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/
-XX:PermSize=512M -XX:MaxPermSize=800m -classpath :::/etc/cloudstack/management:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management/bin/bootstrap.jar:/usr/share/cloudstack-management/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/usr/share/cloudstack-management -Dcatalina.home=/usr/share/cloudstack-management
-Djava.endorsed.dirs= -Djava.io.tmpdir=/usr/share/cloudstack-management/temp -Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap
start
root     28753 28547  0 12:12 pts/1    00:00:00 grep 2941
[root@KiranRhel63-3 init.d]#



The mgmt server is running under the user "cloud".


Next, logged in as cloud user:

[root@KiranRhel63-3 init.d]# su - cloud
-sh-4.1$

-sh-4.1$ ls -ld /var/lib/cloud/mnt/
drwxr-xr-x. 3 cloud cloud 4096 Feb 27 12:06 /var/lib/cloud/mnt/
-sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/
drwxr-xr-x. 3 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/
-sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/7313725325399.503a5754/
drwxr-xr-x. 2 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/7313725325399.503a5754/
-sh-4.1$

Things looked fine till now, but when we tried:

-sh-4.1$ mkdir -p /var/lib/cloud/mnt/VM/7313725325399.503a5767
mkdir: cannot create directory `/var/lib/cloud/mnt/VM/7313725325399.503a5767': Permission
denied
-sh-4.1$


We just changed the last two digits to create a new directory, but are hitting a permission
denied error. Checked whether /var/lib/cloud/mnt/VM or any of the directories in the chain
was already mounted, and they weren't:


-sh-4.1$ df -kh
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_kiranrhel633-lv_root
                       18G  7.8G  8.6G  48% /
tmpfs                 499M     0  499M   0% /dev/shm
/dev/vda1             485M   33M  428M   8% /boot
/mnt/ISO/rhel-server-6.3-x86_64-dvd.iso
                      3.5G  3.5G     0 100% /media
10.147.28.7:/export/home/kiran
                      5.4T  2.5T  3.0T  46% /tmp/sec
10.147.28.7:/export/home/kiran/secondary
                      5.4T  2.5T  3.0T  46% /var/cloudstack/mnt/VM/7614406590488.32f8324
-sh-4.1$


I am not sure what is causing this issue on your server, but it looks like a setup issue.
The mgmt server isn't able to create the mount point under /var/lib/cloud/mnt/ and even manual
creation is failing.

The workaround I can think of to immediately unblock this is for you to create a /mnt/sec
directory as the cloud user, and then mount the nfs secondary directory to /mnt/sec/, and
see how it goes. I believe the mgmt server will proceed beyond the current point of failure
then.

Also, can you please confirm what happens when you run the mgmt server as root? Do you still
see the above issue? I'm guessing that you won't. I know that you should be able to run the
mgmt server as the cloud user - I'm just asking about root to see if it can unblock your testing
user independent functionality. In any case, you must look at why the cloud user cannot create
directories in a directory chain owned by it.


Regards,
Vijay
                
> Unable to Create System Vm's in the VMware Hypervisor setup.
> ------------------------------------------------------------
>
>                 Key: CLOUDSTACK-1228
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1228
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Install and Setup
>    Affects Versions: 4.1.0
>            Reporter: Kiran Koneti
>            Assignee: Venkata Siva Vijayendra Bhamidipati
>            Priority: Blocker
>             Fix For: 4.1.0
>
>         Attachments: catalina.2013-02-11.zip, catalina.zip, management-server-18-2-2013.zip,
management-server.zip
>
>
> Created a Advanced Zone setup using the Latest 4.1 build (http://repo-ccp.citrix.com/releases/ASF/rhel/6.3/4.1/CloudStack-non-OSS-11-rhel6.3.tar.gz)
> While trying to create the system VM's error messages are obserevd.
> Attaching the Management server logs.
> Error message Observed:
> Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.147.28.7/export/home/kiran/s,
templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
> 2013-02-11 23:03:41,530 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.47bfef59
> 2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.648a617f
> 2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.2257f5b0
> 2013-02-11 23:03:41,532 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.3e22e9af
> 2013-02-11 23:03:41,533 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.1e46e9bc
> 2013-02-11 23:03:41,534 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.49fee4a0
> 2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.c1b6a94
> 2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.5604ba42
> 2013-02-11 23:03:41,536 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.3632e0de
> 2013-02-11 23:03:41,537 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.22475368
> 2013-02-11 23:03:41,537 WARN  [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create a mount point
> 2013-02-11 23:03:41,538 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18)
Unable to create mount point for nfs://10.147.28.7/export/home/kiran/s
> 2013-02-11 23:03:41,538 INFO  [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18)
Secondary storage mount point: /mnt/sec
> 2013-02-11 23:03:41,538 INFO  [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18)
Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-02-11 23:03:41,539 DEBUG [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18)
Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-02-11 23:03:41,603 WARN  [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18)
Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"):
java.io.IOException: error=2, No such file or directory
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
>         at com.cloud.utils.script.Script.execute(Script.java:183)
>         at com.cloud.utils.script.Script.execute(Script.java:161)
>         at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:443)
>         at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
>         at java.lang.ProcessImpl.start(ProcessImpl.java:81)
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
>         ... 15 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message