Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Date: Fri, 22 Feb 2013 00:22:14 +0000 (UTC)
From: "Venkata Siva Vijayendra Bhamidipati (JIRA)" <jira@apache.org>
To: cloudstack-dev@incubator.apache.org
Message-ID: <JIRA.12631716.1360584817567.320466.1361492534409@arcas>
In-Reply-To: <JIRA.12631716.1360584817567@arcas>
References: <JIRA.12631716.1360584817567@arcas>
Subject: [jira] [Commented] (CLOUDSTACK-1228) Unable to Create System Vm's
 in the VMware Hypervisor setup.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CLOUDSTACK-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583732#comment-13583732 ] 

Venkata Siva Vijayendra Bhamidipati commented on CLOUDSTACK-1228:
-----------------------------------------------------------------

Hi Kiran,

The mgmt server is attempting to do an mkdir  /var/lib/cloud/mnt/VM/7313725325399.47bfef59 and that is failing for some reason. The mgmt server doesn't log that reason currently. If it fails, the mgmt server code returns a default value of /mnt/sec/ instead. I don't know why it should be doing that. Ideally it should throw an exception and cry foul. Going by the CS installation guide, it should atleast return /mnt/secondary/ and not /mnt/sec/.

Anyway, if the creation of the /var/lib/cloud/mnt/VM directory had gone through correctly, the mgmt server then would have basically mounted the remote NFS mount point (where the ova files exist) to this directory, and would pick up the system template from there. If it fails to locate that system template ova file (which it definitely will fail to because the /var/lib/cloud/mnt/VM directory never got created, so /var/lib/cloud/mnt/VM/xyz/template/tmpl/1/8/<uuidxyz>.ova would not be found), it defaults the template name to routing-8.ova. So looking for a file by name routing-8.ova is futile - it won't exist unless we copy over the <longname>.ova system template under /mnt/sec/template/tmpl/1/8/ as routing-8.ova.

In this case, it's trying to pick it up from /mnt/sec/ which itself doesn't exist and so that will definitely fail anyway.

In short the root cause is that the secondary mount point is not getting created on the mgmt server node. That needs to be investigated. It does not look like a code issue to me at all - it's most likely some config issue on the mgmt server box.


So -

1. Why did the mkdir /var/lib/cloud/mnt/ fail? I strongly suggest starting with a fresh setup like Pradeep suggested, and as root user, checking whether all the nfs settings/options have been set correctly as described in the CS installation guide, with umask set such that umask -S shows: u=rwx,g=rx,o=rx (so set it to 0022), checking whether an mkdir of this directory (/var/lib/cloud/mnt/) by the root user works, and if mounting the nfs directory on this mount point works like suggested by Sateesh, and then removing this directory and starting up the mgmt server to see whether it can create the directory.
2. Why is the mgmt server returning /mnt/sec? I'd remove that. I don't see how it can be of any use. Anyway this won't resolve this issue and isn't a cause. It's a side issue.

Please try out the above and let me know how it goes.

Regards,
Vijay
                
> Unable to Create System Vm's in the VMware Hypervisor setup.
> ------------------------------------------------------------
>
>                 Key: CLOUDSTACK-1228
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1228
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Install and Setup
>    Affects Versions: 4.1.0
>            Reporter: Kiran Koneti
>            Assignee: Venkata Siva Vijayendra Bhamidipati
>            Priority: Blocker
>             Fix For: 4.1.0
>
>         Attachments: catalina.2013-02-11.zip, catalina.zip, management-server-18-2-2013.zip, management-server.zip
>
>
> Created a Advanced Zone setup using the Latest 4.1 build (http://repo-ccp.citrix.com/releases/ASF/rhel/6.3/4.1/CloudStack-non-OSS-11-rhel6.3.tar.gz)
> While trying to create the system VM's error messages are obserevd.
> Attaching the Management server logs.
> Error message Observed:
> Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.147.28.7/export/home/kiran/s, templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
> 2013-02-11 23:03:41,530 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.47bfef59
> 2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.648a617f
> 2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.2257f5b0
> 2013-02-11 23:03:41,532 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.3e22e9af
> 2013-02-11 23:03:41,533 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.1e46e9bc
> 2013-02-11 23:03:41,534 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.49fee4a0
> 2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.c1b6a94
> 2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.5604ba42
> 2013-02-11 23:03:41,536 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.3632e0de
> 2013-02-11 23:03:41,537 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.22475368
> 2013-02-11 23:03:41,537 WARN  [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create a mount point
> 2013-02-11 23:03:41,538 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount point for nfs://10.147.28.7/export/home/kiran/s
> 2013-02-11 23:03:41,538 INFO  [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Secondary storage mount point: /mnt/sec
> 2013-02-11 23:03:41,538 INFO  [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-02-11 23:03:41,539 DEBUG [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-02-11 23:03:41,603 WARN  [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
>         at com.cloud.utils.script.Script.execute(Script.java:183)
>         at com.cloud.utils.script.Script.execute(Script.java:161)
>         at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:443)
>         at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
>         at java.lang.ProcessImpl.start(ProcessImpl.java:81)
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
>         ... 15 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira