Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: domain of guangjian@gmail.com
 designates 209.85.216.45 as permitted sender)
MIME-Version: 1.0
Date: Fri, 12 Apr 2013 10:43:27 +0800
Message-ID: 
 <CAKryD0ZUjJZKum2Fz3urj7okrR1+hMtPYSCFR5u_da_J=P13_A@mail.gmail.com>
Subject: Problem when use Ceph 0.56.4 as primary storage in CloudStack 3.0.2
From: Guangjian Liu <guangjian@gmail.com>
To: users@cloudstack.apache.org
Content-Type: multipart/alternative; boundary=047d7b2e7ad22990b604da20de3b

--047d7b2e7ad22990b604da20de3b
Content-Type: text/plain; charset=ISO-8859-1

Problem when use Ceph 0.56.4 as primary storage in CloudStack 3.0.2, ssvm
and cpvm can not start up in normal. If i should migartion cs to 4.0? as i
know cs 4.0 support ceph RBD.
*
*
*Environment:*
Server 1: Cloudstack 3.0.2 in Redhat RHEL 6.2 Kernel 2.6.32
Server 2: KVM in Redhat RHEL 6.2 Kernel  2.6.32, with Ceph 0.56.4

*Ceph Mount in Server2:*
[root@RDR02S01 ceph]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
ceph-fuse            143233024   3371008 139862016   3% /cloudfs

ps
root     29885     1  0 Apr10 ?        00:01:06 ceph-fuse -m
10.0.0.41:6789/cloudfs/
[root@RDR02S01 ceph]# ll /cloudfs/zone_kvm_bas/
total 256

*Case:*
Create basic zone with kvm, use ceph for primary storage.
Mount in CS:
Type: SharedMountPoint
Path: /cloudfs/zone_kvm_bas

*Problem:*
Basic Zone created success, but ssvm and cpvm can not start up, always down
and restart.

exception in log as below:
2013-04-11 00:03:51,052 DEBUG [agent.transport.Request] (secstorage-1:null)
Seq 27-1926955018: Sending  { Cmd , MgmtId: 133846799640, via: 27, Ver: v1,
Flags: 100111,
[{"storage.PrimaryStorageDownloadCommand":{"localPath":"/mnt/9fa1f231-3fda-3a0f-adeb-d8d53871243d","poolUuid":"4c5fb9ed-669f-41d7-9eba-b4213e605e29","poolId":243,"secondaryStorageUrl":"nfs://
10.0.0.2/backup/secondary_kvm
","primaryStorageUrl":"nfs://localhost/cloudfs/zone_kvm_bas","url":"nfs://
10.0.0.2/backup/secondary_kvm/template/tmpl/1/3/","format":"QCOW2","accountId":1,"name":"routing-3","wait":10800}}]
}
2013-04-11 00:03:51,057 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleaning up resources for the vm
VM[ConsoleProxy|v-3657-VM] in Starting state
2013-04-11 00:03:51,060 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 27-1926955019: Waiting for Seq 1926955018
Scheduling:  { Cmd , MgmtId: 133846799640, via: 27, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"v-3657-VM","wait":0}}] }
2013-04-11 00:03:51,113 DEBUG [agent.transport.Request]
(AgentManager-Handler-10:null) Seq 27-1926955018: Processing:  { Ans: ,
MgmtId: 133846799640, via: 27, Ver: v1, Flags: 110,
[{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)\n\tat
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)\n\tat
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)\n\tat
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)\n\tat
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)\n\tat
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:517)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)\n\tat
com.cloud.utils.nio.Task.run(Task.java:79)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-04-11 00:03:51,113 WARN  [cloud.vm.VirtualMachineManagerImpl]
(AgentManager-Handler-10:null) Cleanup failed due to
java.lang.NullPointerException
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)
        at
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)
        at
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)
        at
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)
        at com.cloud.agent.Agent.processRequest(Agent.java:517)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
        at com.cloud.utils.nio.Task.run(Task.java:79)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)

2013-04-11 00:03:51,113 DEBUG [agent.transport.Request] (secstorage-1:null)
Seq 27-1926955018: Received:  { Ans: , MgmtId: 133846799640, via: 27, Ver:
v1, Flags: 110, { Answer } }
2013-04-11 00:03:51,114 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-10:null) Seq 27-1926955019: Sending now.  is current
sequence.
2013-04-11 00:03:51,114 WARN  [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Cleanup failed due to java.lang.NullPointerException
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)
        at
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)
        at
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)
        at
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)
        at com.cloud.agent.Agent.processRequest(Agent.java:517)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
        at com.cloud.utils.nio.Task.run(Task.java:79)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)

2013-04-11 00:03:51,114 DEBUG [agent.manager.AgentManagerImpl]
(secstorage-1:null) Details from executing class
com.cloud.agent.api.storage.PrimaryStorageDownloadCommand:
java.lang.NullPointerException
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)
        at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)
        at
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)
        at
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)
        at
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)
        at com.cloud.agent.Agent.processRequest(Agent.java:517)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
        at com.cloud.utils.nio.Task.run(Task.java:79)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)

2013-04-11 00:03:51,115 ERROR [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Failed to start instance
VM[SecondaryStorageVm|s-3656-VM]
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.storage.PrimaryStorageDownloadAnswer
        at
com.cloud.template.TemplateManagerImpl.prepareTemplateForCreate(TemplateManagerImpl.java:659)
        at
com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30)
        at
com.cloud.storage.StorageManagerImpl.createVolume(StorageManagerImpl.java:3065)
        at
com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:2981)
        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:743)
        at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:461)
        at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:454)
        at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:243)
        at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:670)
        at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
        at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:115)
        at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:46)
        at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:102)
        at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:30)
        at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:79)
        at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:69)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-04-11 00:03:51,160 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Cleaning up resources for the vm
VM[SecondaryStorageVm|s-3656-VM] in Starting state
2013-04-11 00:03:51,162 DEBUG [agent.transport.Request] (secstorage-1:null)
Seq 27-1926955020: Waiting for Seq 1926955019 Scheduling:  { Cmd , MgmtId:
133846799640, via: 27, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"s-3656-VM","wait":0}}] }
2013-04-11 00:03:51,395 DEBUG [agent.transport.Request]
(AgentManager-Handler-11:null) Seq 27-1926955019: Processing:  { Ans: ,
MgmtId: 133846799640, via: 27, Ver: v1, Flags: 110,
[{"StopAnswer":{"vncPort":0,"bytesSent":0,"bytesReceived":0,"result":true,"wait":0}}]
}
2013-04-11 00:03:51,395 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(AgentManager-Handler-11:null) Cleanup succeeded. Details null
2013-04-11 00:03:51,395 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 27-1926955019: Received:  { Ans: , MgmtId:
133846799640, via: 27, Ver: v1, Flags: 110, { StopAnswer } }


-- 
Guangjian

--047d7b2e7ad22990b604da20de3b--