cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Graham <tgra...@tgraham.us>
Subject Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand
Date Wed, 09 Oct 2013 03:11:10 GMT
Might be this one:

* https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting

Travis

On Oct 8, 2013, at 11:04 PM, Mike Tutkowski <mike.tutkowski@solidfire.com> wrote:

> CS MS can ping the SSVM's public IP address.
> 
> SSVM can ping CS MS.
> 
> I think we had a nice debug document for SSVM mentioned on a CS
> e-mail...I'll try to find it.
> 
> 
> On Tue, Oct 8, 2013 at 8:29 PM, Marcus Sorensen <shadowsor@gmail.com> wrote:
> 
>> You could start by trying to ping them from mgmt server, or trying to VNC
>> to them if the OS isn't up. I'm assuming you prepped a current system vm
>> template (maybe you're even reusing a secondary storage). If you can ping
>> their 169.254 addresses, you can "cloudstack-ssh 169.254.x.x" as root on
>> the kvm host to get in
>> On Oct 8, 2013 7:31 PM, "Mike Tutkowski" <mike.tutkowski@solidfire.com>
>> wrote:
>> 
>>> Looks like I'll have to dig up that doc on debugging SSVM...it's been
>>> about 30 mins since the VMs came up.
>>> 
>>> 
>>> On Tue, Oct 8, 2013 at 7:29 PM, Marcus Sorensen <shadowsor@gmail.com>wrote:
>>> 
>>>> You may just have to wait a bit. That means the VMS are started but the
>>>> services aren't up. Usually takes 5-10 min to get both up in my fusion vm.
>>>> If that doesn't get better you may have to log into the system VMS and see
>>>> what's going on.
>>>> On Oct 8, 2013 7:25 PM, "Mike Tutkowski" <mike.tutkowski@solidfire.com>
>>>> wrote:
>>>> 
>>>>> Doh! I didn't restart the necessary service after exporting my NFS
>>>>> share.
>>>>> 
>>>>> The system VMs come up now; however, SSVM has a dash (-) for its Agent
>>>>> State and the CS MS console keeps printing out the following:
>>>>> 
>>>>> INFO  [o.a.c.s.e.DefaultEndPointSelector]
>>>>> (StatsCollector-3:ctx-018adc41) No running ssvm is found, so command
will
>>>>> be sent to LocalHostEndPoint
>>>>> 
>>>>> 
>>>>> On Tue, Oct 8, 2013 at 4:35 PM, Marcus Sorensen <shadowsor@gmail.com>wrote:
>>>>> 
>>>>>> Can you mount the secondary storage from your KVM host?
>>>>>> On Oct 8, 2013 4:01 PM, "Mike Tutkowski" <mike.tutkowski@solidfire.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Although the host is added to KVM, I do see the following issues
in
>>>>>>> the CS MS console (any thoughts on this?):
>>>>>>> 
>>>>>>> WARN  [c.c.u.d.Merovingian2] (secstorage-1:ctx-c1c573ee) Was
unable
>>>>>>> to find lock for the key template_spool_ref2 and thread id 2049868806
>>>>>>> INFO  [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-c1c573ee)
>>>>>>> Unable to contact resource.
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:1] is unreachable: Unable to create
>>>>>>> Vol[1|vm=1|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>>>>>>> org.libvirt.LibvirtException: internal error Child process (/bin/mount
>>>>>>> 192.168.233.10:/mnt/secondary/template/tmpl/1/3
>>>>>>> /mnt/334b3c4e-764b-362a-be2c-ebe8c490d0a9) status unexpected:
exit status 32
>>>>>>> at
>>>>>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1027)
>>>>>>> at
>>>>>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1069)
>>>>>>> at
>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:830)
>>>>>>> at
>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:649)
>>>>>>> at
>>>>>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
>>>>>>> at
>>>>>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:693)
>>>>>>> at
>>>>>>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
>>>>>>> at
>>>>>>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
>>>>>>> at
>>>>>>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
>>>>>>> at
>>>>>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
>>>>>>> at
>>>>>>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
>>>>>>> at
>>>>>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
>>>>>>> at
>>>>>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
>>>>>>> at
>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:724)
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Oct 8, 2013 at 3:58 PM, Mike Tutkowski <
>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>> 
>>>>>>>> So...got some good news:
>>>>>>>> 
>>>>>>>> I spent a couple hours setting up a KVM environment on Ubuntu
>>>>>>>> 12.04.1 from scratch (Installing SSH, Open iSCSI, Java 7,
KVM, Git,
>>>>>>>> CloudStack, CloudStack DEBs, KVM system template, etc.) and
I can now add
>>>>>>>> this KVM host to CloudStack (on a related note, no errors
in agent.err
>>>>>>>> either).
>>>>>>>> 
>>>>>>>> I have no idea what is messed up with my old KVM install
on Ubuntu,
>>>>>>>> but the new one works.
>>>>>>>> 
>>>>>>>> That being the case, I can close out the JIRA ticket I logged
a
>>>>>>>> while back and start integrating your code into mine.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Oct 7, 2013 at 7:46 PM, Mike Tutkowski <
>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>> 
>>>>>>>>> Right...yeah, I didn't mean we'd commit to 4.2, but maybe
I should
>>>>>>>>> work off of 4.2 since master seems to be un-stable in
this regard.
>>>>>>>>> 
>>>>>>>>> I plan to set up a machine in the lab tomorrow with Ubuntu
12.04
>>>>>>>>> from scratch to see if it works when I start clean, but
- if it doesn't - I
>>>>>>>>> should just use 4.2 for development.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, Oct 7, 2013 at 7:05 PM, Marcus Sorensen <
>>>>>>>>> shadowsor@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> We can't. This patch will never see 4.2. You can
still start
>>>>>>>>>> working on your plugin on 4.2, but the change represented
by this patch can
>>>>>>>>>> only go into master.
>>>>>>>>>> On Oct 7, 2013 5:01 PM, "Mike Tutkowski" <
>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> So, now that I'm getting back to this, do you
think I should just
>>>>>>>>>>> try to make this work with 4.2 (like we originally
talked about)?
>>>>>>>>>>> 
>>>>>>>>>>> I updated again from master, rebuilt, redeployed
DEBs and still
>>>>>>>>>>> get this JNA error message:
>>>>>>>>>>> 
>>>>>>>>>>> log4j:WARN No appenders could be found for logger
>>>>>>>>>>> (org.apache.commons.httpclient.params.DefaultHttpParams).
>>>>>>>>>>> log4j:WARN Please initialize the log4j system
properly.
>>>>>>>>>>> log4j:WARN See
>>>>>>>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more
>>>>>>>>>>> info.
>>>>>>>>>>> java.lang.reflect.InvocationTargetException
>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>>>> at
>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>>> at
>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
>>>>>>>>>>> Caused by: java.lang.UnsatisfiedLinkError: Can't
obtain
>>>>>>>>>>> updateLastError method for class com.sun.jna.Native
>>>>>>>>>>> at com.sun.jna.Native.initIDs(Native Method)
>>>>>>>>>>> at com.sun.jna.Native.<clinit>(Native.java:139)
>>>>>>>>>>> at org.libvirt.jna.Libvirt.<clinit>(Unknown
Source)
>>>>>>>>>>> at org.libvirt.Library.<clinit>(Unknown
Source)
>>>>>>>>>>> at org.libvirt.Connect.<init>(Unknown Source)
>>>>>>>>>>> at
>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:44)
>>>>>>>>>>> at
>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:37)
>>>>>>>>>>> at
>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:733)
>>>>>>>>>>> at com.cloud.agent.Agent.<init>(Agent.java:161)
>>>>>>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
>>>>>>>>>>> at
>>>>>>>>>>> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:370)
>>>>>>>>>>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
>>>>>>>>>>> at com.cloud.agent.AgentShell.start(AgentShell.java:448)
>>>>>>>>>>> ... 5 more
>>>>>>>>>>> Cannot start daemon
>>>>>>>>>>> Service exit with a return value of 5
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Oct 7, 2013 at 2:31 PM, Mike Tutkowski
<
>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Sure, that's a good plan.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'll get to it.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Oct 7, 2013 at 2:29 PM, Marcus Sorensen
<
>>>>>>>>>>>> shadowsor@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I know you mentioned you might need some
minor changes to it,
>>>>>>>>>>>>> as well as other minor changes just for
master (attach volume switched to
>>>>>>>>>>>>> pool vs adapter or something). My hope
was that you would be able to send
>>>>>>>>>>>>> an update that works for your plugin
on master, I'll test against existing
>>>>>>>>>>>>> libvirtd storage and apply it.
>>>>>>>>>>>>> On Oct 7, 2013 1:49 PM, "Mike Tutkowski"
<
>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>   This is an automatically generated
e-mail. To reply,
>>>>>>>>>>>>>> visit: https://reviews.apache.org/r/14381/
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This looks reasonable to me, Marcus.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> When do you think you might start
the process of getting this into master?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - Mike Tutkowski
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On September 30th, 2013, 5:14 p.m.
UTC, Marcus Sorensen wrote:
>>>>>>>>>>>>>>  Review request for cloudstack, edison
su and Mike Tutkowski.
>>>>>>>>>>>>>> By Marcus Sorensen.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> *Updated Sept. 30, 2013, 5:14 p.m.*
>>>>>>>>>>>>>> *Repository: * cloudstack-git
>>>>>>>>>>>>>> Description
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> With custom storage plugins comes
the need to prep the KVM host prior to utilizing the disks. e.g. an iscsi initiator needs
to log into the target and scan for the lun before it can be used on the host. This patch
is an example I developed against 4.2, minor changes may be necessary to apply to master,
but I want to share with others who are working on storage so they can ensure it works for
them. Please tweak as you see fit.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MigrateCommand: pass vmTO object
so we can see which disks/storage pool types belong to the vm when migrating a VM. This facilitates
being able to call disconnectPhysicalDisksViaVmSpec
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> VirtualMachineManagerImpl: pass VirtualMachineTO
when migrating so that we can see which disks belong to the VM and what storage pools/adaptors
should be used
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> LibvirtComputingResource: add calls
KVMStoragePoolManager's connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec calls
where appropriate (when starting a vm, migrating a vm). Ensure that we create 'raw' format
XML disk definitions when the storage format is RAW. Move cleanupDisk logic to storage adaptors
so that each adaptor type can clean up its disks in is own way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> KVMStoragePoolManager:  add connectPhysicalDisk,
disconnectPhysicalDisk, connectPhysicalDiskViaVmSpec, disconnectPhysicalDiskViaVmSpec, disconnectPhysicalDiskByPath.
These all call the specific StorageAdaptor's connectPhysicalDisk, disconnectPhysicalDisk,
or disconnectPhysicalDiskByPath calls.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> KVMStorageProcessor: Call connectPhysicalDisk/disconnectPhysicalDisk
on the storage adaptor. Whether or not this is implemented is up to the storage adaptor.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> LibvirtStorageAdaptor: implement
dummy connectPhysicalDisk/disconnectPhysicalDisk, move cleanupDisk logic from LibvirtComputingResource
to disconnectPhysicalDiskByPath
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> StorageAdaptor: define connectPhysicalDisk/disconnectPhysicalDisk/disconnectPhysicalDiskByPath
in the interface
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>  Testing
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Basic testing with my storage adaptor
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>  Diffs
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>   - core/src/com/cloud/agent/api/MigrateCommand.java
>>>>>>>>>>>>>>   (5042b8c)
>>>>>>>>>>>>>>   - plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
>>>>>>>>>>>>>>   (3ee811f)
>>>>>>>>>>>>>>   - plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java
>>>>>>>>>>>>>>   (e09c9ba)
>>>>>>>>>>>>>>   - plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
>>>>>>>>>>>>>>   (c69f9b0)
>>>>>>>>>>>>>>   - plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
>>>>>>>>>>>>>>   (123a9f1)
>>>>>>>>>>>>>>   - plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java
>>>>>>>>>>>>>>   (4956d8d)
>>>>>>>>>>>>>>   - server/src/com/cloud/vm/VirtualMachineManagerImpl.java
>>>>>>>>>>>>>>   (d46bbb0)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> View Diff <https://reviews.apache.org/r/14381/diff/>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>>> *™*
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>> *™*
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> *Mike Tutkowski*
>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>> o: 303.746.7302
>>>>>>>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>> *™*
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> *Mike Tutkowski*
>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>> o: 303.746.7302
>>>>>>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>> *™*
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> *Mike Tutkowski*
>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>> o: 303.746.7302
>>>>>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>> *™*
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>>>> *™*
>>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>>> *™*
>>> 
>> 
> 
> 
> -- 
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message