Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7A05410078 for ; Fri, 11 Oct 2013 01:57:01 +0000 (UTC) Received: (qmail 88387 invoked by uid 500); 11 Oct 2013 01:57:01 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 88349 invoked by uid 500); 11 Oct 2013 01:57:01 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 88338 invoked by uid 99); 11 Oct 2013 01:57:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Oct 2013 01:57:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of mike.tutkowski@solidfire.com designates 209.85.217.176 as permitted sender) Received: from [209.85.217.176] (HELO mail-lb0-f176.google.com) (209.85.217.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Oct 2013 01:56:56 +0000 Received: by mail-lb0-f176.google.com with SMTP id y6so2859245lbh.35 for ; Thu, 10 Oct 2013 18:56:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=VjG5TtP4nIPZyyRSPd2CGx7N9PI5i1DbEwWRzNefkEA=; b=a4c9ROz/WfJppkIf6f23Cbqh8+ihXvc61w9xjTIHvrBZNkowDG7LThpoBDydpfydx8 W+zQ0tPBCafUw5iHgK96Dlti0OW3uJA44gl2ZEIdJ/hiTDjjxbEalHMxDV+gLn+a6eua gSRq47j/D1wQ8yEzMUzD3z3yij8PpB+yopV2eudU17u2Ua2JQCRhD0+S87p7hzOCvQqs fCIqNlgs0QdKefGFUUqzLpIkSlYVmym5plcA4faKa4CDZge5RtX2wlb89/Gx0a1Aw0c+ os8Wmb+EBuaOXiNJIGLd5e75t4rf7Oq5YzmtCWcO6jAwcikpt/BnUCp87r2xVNIF+iv0 QqXQ== X-Gm-Message-State: ALoCoQneux9xlw2G/z3pULLY9kf5qP9MrzDZm2b4lozVC/BqmHgC+nYvk6yu0IDp7yIuboJRHtDm MIME-Version: 1.0 X-Received: by 10.152.120.5 with SMTP id ky5mr14108328lab.18.1381456594419; Thu, 10 Oct 2013 18:56:34 -0700 (PDT) Received: by 10.114.184.17 with HTTP; Thu, 10 Oct 2013 18:56:34 -0700 (PDT) In-Reply-To: References: <20130930171409.28587.79669@reviews.apache.org> <20131007194929.20898.27320@reviews.apache.org> Date: Thu, 10 Oct 2013 19:56:34 -0600 Message-ID: Subject: Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand From: Mike Tutkowski To: Marcus Sorensen Cc: "dev@cloudstack.apache.org" Content-Type: multipart/alternative; boundary=089e01227ca495525004e86d6d08 X-Virus-Checked: Checked by ClamAV on apache.org --089e01227ca495525004e86d6d08 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I made a bunch of updates, Marcus: https://github.com/mike-tutkowski/incubator-cloudstack/commit/c13442eaa228a= 5ae37d3b13f6c98636de158a908 On Thu, Oct 10, 2013 at 4:40 PM, Mike Tutkowski < mike.tutkowski@solidfire.com> wrote: > Hey Marcus, > > I haven't had a chance to test this yet, but I thought you might be > interested in the merge of your work with mine. > > I've also replaced the direct CHAP parameters with a Map > (which currently only stores the CHAP credentials, but could be used for > more in the future). > > I still have a bit of work to do around the getPhysicalDisk method, but > this should give you a good idea of where we're at: > > > https://github.com/mike-tutkowski/incubator-cloudstack/commit/30433a20d17= 07ea1870024e80732bcfdbd64f69a > > Talk to you later > > > On Wed, Oct 9, 2013 at 3:52 PM, Mike Tutkowski < > mike.tutkowski@solidfire.com> wrote: > >> I've got the code all merged, by the way. >> >> I should be able to start in on testing soon (tonight or tomorrow). >> >> >> On Wed, Oct 9, 2013 at 3:51 PM, Mike Tutkowski < >> mike.tutkowski@solidfire.com> wrote: >> >>> Will do...I'll remove it (there is similar code in two methods). >>> >>> >>> On Wed, Oct 9, 2013 at 3:49 PM, Marcus Sorensen wr= ote: >>> >>>> Yeah, that looks like leftovers from refactoring our 4.1 code into 4.2 >>>> and making it something more generic. It looks like it could be remove= d. >>>> >>>> >>>> On Wed, Oct 9, 2013 at 3:40 PM, Mike Tutkowski < >>>> mike.tutkowski@solidfire.com> wrote: >>>> >>>>> Hey Marcus, >>>>> >>>>> I'm merging your changes into mine. >>>>> >>>>> It looks like I can remove your call to getPhysicalDisk (below) (and >>>>> the associated KVMPhysicalDisk variable, as well). >>>>> >>>>> I assume you would never want to produce any side effects in your >>>>> getPhysicalDisk implementation, right? Seems like that would be unint= uitive. >>>>> >>>>> public boolean connectPhysicalDisksViaVmSpec(VirtualMachineTO >>>>> vmSpec) { >>>>> >>>>> boolean result =3D false; >>>>> >>>>> >>>>> final String vmName =3D vmSpec.getName(); >>>>> >>>>> >>>>> List disks =3D Arrays.asList(vmSpec.getDisks()); >>>>> >>>>> >>>>> for (DiskTO disk : disks) { >>>>> >>>>> KVMPhysicalDisk physicalDisk =3D null; >>>>> >>>>> KVMStoragePool pool =3D null; >>>>> >>>>> >>>>> if (disk.getType() !=3D Volume.Type.ISO) { >>>>> >>>>> VolumeObjectTO vol =3D (VolumeObjectTO) disk.getData(= ); >>>>> >>>>> PrimaryDataStoreTO store =3D (PrimaryDataStoreTO) >>>>> vol.getDataStore(); >>>>> >>>>> >>>>> pool =3D getStoragePool(store.getPoolType(), >>>>> store.getUuid()); >>>>> >>>>> physicalDisk =3D pool.getPhysicalDisk(vol.getPath()); >>>>> >>>>> >>>>> StorageAdaptor adaptor =3D >>>>> getStorageAdaptor(pool.getType()); >>>>> >>>>> >>>>> result =3D adaptor.connectPhysicalDisk(vol.getPath(), >>>>> pool); >>>>> >>>>> >>>>> if (! result) { >>>>> >>>>> s_logger.error("Failed to connect disks via vm >>>>> spec for vm:" + vmName + " volume:" + vol.toString()); >>>>> >>>>> >>>>> return result; >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> return result; >>>>> >>>>> } >>>>> >>>>> >>>>> On Wed, Oct 9, 2013 at 12:01 AM, Mike Tutkowski < >>>>> mike.tutkowski@solidfire.com> wrote: >>>>> >>>>>> Excellent...that template worked like a charm. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> On Tue, Oct 8, 2013 at 11:50 PM, Mike Tutkowski < >>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>> >>>>>>> Perfect >>>>>>> >>>>>>> Almost done downloading...I'll give it a try in a moment. >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 8, 2013 at 11:47 PM, Marcus Sorensen < >>>>>>> shadowsor@gmail.com> wrote: >>>>>>> >>>>>>>> Not password enabled, root password is just 'password'. I mainly >>>>>>>> just use it for devcloud-kvm testing. >>>>>>>> On Oct 8, 2013 11:45 PM, "Mike Tutkowski" < >>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>> >>>>>>>>> Great - thanks! >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 8, 2013 at 11:38 PM, Marcus Sorensen < >>>>>>>>> shadowsor@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Use my tiny centos image. I'm not sure what's up with that cruft= y >>>>>>>>>> old default centos template. >>>>>>>>>> >>>>>>>>>> Register this qcow2 template: >>>>>>>>>> http://marcus.mlsorensen.com/cloudstack-extras/tiny-centos-63.qc= ow2 >>>>>>>>>> >>>>>>>>>> Needs a service offering with at least 192MB to run. >>>>>>>>>> On Oct 8, 2013 11:36 PM, "Mike Tutkowski" < >>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>> >>>>>>>>>>> Perhaps you might know something about this, Marcus. >>>>>>>>>>> >>>>>>>>>>> My instance suffers a Kernel panic while booting up. >>>>>>>>>>> >>>>>>>>>>> I'm just using the built-in KVM template (CentOS 5.5(64-bit) no >>>>>>>>>>> GUI (KVM)) with 1 CPU and 512 MB memory. >>>>>>>>>>> >>>>>>>>>>> http://i.imgur.com/QuPH2Ub.png >>>>>>>>>>> >>>>>>>>>>> I tried to just use an ISO instead, but apparently that >>>>>>>>>>> functionality is broken, as well (related to Disk Offerings). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 8, 2013 at 10:39 PM, Mike Tutkowski < >>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> OK, all is good now. >>>>>>>>>>>> >>>>>>>>>>>> I have both system VMs up and running and the Agent States rea= d >>>>>>>>>>>> as "Up," as well. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Oct 8, 2013 at 9:50 PM, Mike Tutkowski < >>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I believe we've been down this road before: >>>>>>>>>>>>> >>>>>>>>>>>>> 2013-10-09 03:47:41,281 ERROR [cloud.agent.AgentShell] >>>>>>>>>>>>> (main:null) Unable to start agent: Resource class not found: >>>>>>>>>>>>> com.cloud.storage.resource.PremiumSecondaryStorageResource du= e to: >>>>>>>>>>>>> java.lang.ClassNotFoundException: >>>>>>>>>>>>> com.cloud.storage.resource.PremiumSecondaryStorageResource >>>>>>>>>>>>> >>>>>>>>>>>>> The solution was to compile without -Dnoredist. >>>>>>>>>>>>> >>>>>>>>>>>>> I will try that now. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Oct 8, 2013 at 9:36 PM, Marcus Sorensen < >>>>>>>>>>>>> shadowsor@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> You may be able to find a stack trace for the java process i= n >>>>>>>>>>>>>> /var/log/cloud or the messages file, on the system vm. >>>>>>>>>>>>>> On Oct 8, 2013 9:21 PM, "Mike Tutkowski" < >>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Interesting...I ran the following: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /usr/local/cloud/systemvm/ssvm-check.sh >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It says the Java process is not running. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is the KVM system template I'm using: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://download.cloud.com/templates/4.2/systemvmtemplate-20= 13-06-12-master-kvm.qcow2.bz2 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I just picked the one that was referenced in the VM_Templat= e >>>>>>>>>>>>>>> table in 4.3. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Oct 8, 2013 at 9:05 PM, Mike Tutkowski < >>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Found it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As an FYI, this is the doc I was referring to: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSV= M%2C+templates%2C+Secondary+storage+troubleshooting >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Oct 8, 2013 at 9:04 PM, Mike Tutkowski < >>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CS MS can ping the SSVM's public IP address. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> SSVM can ping CS MS. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think we had a nice debug document for SSVM mentioned >>>>>>>>>>>>>>>>> on a CS e-mail...I'll try to find it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Oct 8, 2013 at 8:29 PM, Marcus Sorensen < >>>>>>>>>>>>>>>>> shadowsor@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> You could start by trying to ping them from mgmt server, >>>>>>>>>>>>>>>>>> or trying to VNC to them if the OS isn't up. I'm assumin= g you prepped a >>>>>>>>>>>>>>>>>> current system vm template (maybe you're even reusing a = secondary storage). >>>>>>>>>>>>>>>>>> If you can ping their 169.254 addresses, you can "clouds= tack-ssh >>>>>>>>>>>>>>>>>> 169.254.x.x" as root on the kvm host to get in >>>>>>>>>>>>>>>>>> On Oct 8, 2013 7:31 PM, "Mike Tutkowski" < >>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Looks like I'll have to dig up that doc on debugging >>>>>>>>>>>>>>>>>>> SSVM...it's been about 30 mins since the VMs came up. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Oct 8, 2013 at 7:29 PM, Marcus Sorensen < >>>>>>>>>>>>>>>>>>> shadowsor@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> You may just have to wait a bit. That means the VMS ar= e >>>>>>>>>>>>>>>>>>>> started but the services aren't up. Usually takes 5-10= min to get both up >>>>>>>>>>>>>>>>>>>> in my fusion vm. If that doesn't get better you may ha= ve to log into the >>>>>>>>>>>>>>>>>>>> system VMS and see what's going on. >>>>>>>>>>>>>>>>>>>> On Oct 8, 2013 7:25 PM, "Mike Tutkowski" < >>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Doh! I didn't restart the necessary service after >>>>>>>>>>>>>>>>>>>>> exporting my NFS share. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The system VMs come up now; however, SSVM has a dash >>>>>>>>>>>>>>>>>>>>> (-) for its Agent State and the CS MS console keeps p= rinting out the >>>>>>>>>>>>>>>>>>>>> following: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> INFO [o.a.c.s.e.DefaultEndPointSelector] >>>>>>>>>>>>>>>>>>>>> (StatsCollector-3:ctx-018adc41) No running ssvm is fo= und, so command will >>>>>>>>>>>>>>>>>>>>> be sent to LocalHostEndPoint >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 8, 2013 at 4:35 PM, Marcus Sorensen < >>>>>>>>>>>>>>>>>>>>> shadowsor@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Can you mount the secondary storage from your KVM >>>>>>>>>>>>>>>>>>>>>> host? >>>>>>>>>>>>>>>>>>>>>> On Oct 8, 2013 4:01 PM, "Mike Tutkowski" < >>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Although the host is added to KVM, I do see the >>>>>>>>>>>>>>>>>>>>>>> following issues in the CS MS console (any thoughts= on this?): >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> WARN [c.c.u.d.Merovingian2] >>>>>>>>>>>>>>>>>>>>>>> (secstorage-1:ctx-c1c573ee) Was unable to find lock= for the key >>>>>>>>>>>>>>>>>>>>>>> template_spool_ref2 and thread id 2049868806 >>>>>>>>>>>>>>>>>>>>>>> INFO [c.c.v.VirtualMachineManagerImpl] >>>>>>>>>>>>>>>>>>>>>>> (secstorage-1:ctx-c1c573ee) Unable to contact resou= rce. >>>>>>>>>>>>>>>>>>>>>>> com.cloud.exception.StorageUnavailableException: >>>>>>>>>>>>>>>>>>>>>>> Resource [StoragePool:1] is unreachable: Unable to = create >>>>>>>>>>>>>>>>>>>>>>> Vol[1|vm=3D1|ROOT]:com.cloud.utils.exception.CloudR= untimeException: >>>>>>>>>>>>>>>>>>>>>>> org.libvirt.LibvirtException: internal error Child = process (/bin/mount >>>>>>>>>>>>>>>>>>>>>>> 192.168.233.10:/mnt/secondary/template/tmpl/1/3 >>>>>>>>>>>>>>>>>>>>>>> /mnt/334b3c4e-764b-362a-be2c-ebe8c490d0a9) status u= nexpected: exit status 32 >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.engine.orchestration.VolumeOr= chestrator.recreateVolume(VolumeOrchestrator.java:1027) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.engine.orchestration.VolumeOr= chestrator.prepare(VolumeOrchestrator.java:1069) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart= (VirtualMachineManagerImpl.java:830) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart= (VirtualMachineManagerImpl.java:649) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.storage.secondary.SecondaryStorageManager= Impl.startSecStorageVm(SecondaryStorageManagerImpl.java:261) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.storage.secondary.SecondaryStorageManager= Impl.allocCapacity(SecondaryStorageManagerImpl.java:693) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.storage.secondary.SecondaryStorageManager= Impl.expandPool(SecondaryStorageManagerImpl.java:1265) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.secstorage.PremiumSecondaryStorageManager= Impl.scanPool(PremiumSecondaryStorageManagerImpl.java:123) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.secstorage.PremiumSecondaryStorageManager= Impl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmL= oadScanner.java:101) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemV= mLoadScanner.java:33) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(System= VmLoadScanner.java:78) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(Sys= temVmLoadScanner.java:71) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContex= tRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultM= anagedContext$1.call(DefaultManagedContext.java:56) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultM= anagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultM= anagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContex= tRunnable.run(ManagedContextRunnable.java:46) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call= (Executors.java:471) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureT= ask.java:304) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$Sc= heduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$Sc= heduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(T= hreadPoolExecutor.java:1145) >>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(= ThreadPoolExecutor.java:615) >>>>>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:724) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 8, 2013 at 3:58 PM, Mike Tutkowski < >>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> So...got some good news: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I spent a couple hours setting up a KVM environmen= t >>>>>>>>>>>>>>>>>>>>>>>> on Ubuntu 12.04.1 from scratch (Installing SSH, Op= en iSCSI, Java 7, KVM, >>>>>>>>>>>>>>>>>>>>>>>> Git, CloudStack, CloudStack DEBs, KVM system templ= ate, etc.) and I can now >>>>>>>>>>>>>>>>>>>>>>>> add this KVM host to CloudStack (on a related note= , no errors in agent.err >>>>>>>>>>>>>>>>>>>>>>>> either). >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I have no idea what is messed up with my old KVM >>>>>>>>>>>>>>>>>>>>>>>> install on Ubuntu, but the new one works. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> That being the case, I can close out the JIRA >>>>>>>>>>>>>>>>>>>>>>>> ticket I logged a while back and start integrating= your code into mine. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 7, 2013 at 7:46 PM, Mike Tutkowski < >>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Right...yeah, I didn't mean we'd commit to 4.2, >>>>>>>>>>>>>>>>>>>>>>>>> but maybe I should work off of 4.2 since master s= eems to be un-stable in >>>>>>>>>>>>>>>>>>>>>>>>> this regard. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I plan to set up a machine in the lab tomorrow >>>>>>>>>>>>>>>>>>>>>>>>> with Ubuntu 12.04 from scratch to see if it works= when I start clean, but - >>>>>>>>>>>>>>>>>>>>>>>>> if it doesn't - I should just use 4.2 for develop= ment. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 7, 2013 at 7:05 PM, Marcus Sorensen < >>>>>>>>>>>>>>>>>>>>>>>>> shadowsor@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> We can't. This patch will never see 4.2. You can >>>>>>>>>>>>>>>>>>>>>>>>>> still start working on your plugin on 4.2, but t= he change represented by >>>>>>>>>>>>>>>>>>>>>>>>>> this patch can only go into master. >>>>>>>>>>>>>>>>>>>>>>>>>> On Oct 7, 2013 5:01 PM, "Mike Tutkowski" < >>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> So, now that I'm getting back to this, do you >>>>>>>>>>>>>>>>>>>>>>>>>>> think I should just try to make this work with = 4.2 (like we originally >>>>>>>>>>>>>>>>>>>>>>>>>>> talked about)? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I updated again from master, rebuilt, redeploye= d >>>>>>>>>>>>>>>>>>>>>>>>>>> DEBs and still get this JNA error message: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> log4j:WARN No appenders could be found for >>>>>>>>>>>>>>>>>>>>>>>>>>> logger (org.apache.commons.httpclient.params.De= faultHttpParams). >>>>>>>>>>>>>>>>>>>>>>>>>>> log4j:WARN Please initialize the log4j system >>>>>>>>>>>>>>>>>>>>>>>>>>> properly. >>>>>>>>>>>>>>>>>>>>>>>>>>> log4j:WARN See >>>>>>>>>>>>>>>>>>>>>>>>>>> http://logging.apache.org/log4j/1.2/faq.html#no= configfor more info. >>>>>>>>>>>>>>>>>>>>>>>>>>> java.lang.reflect.InvocationTargetException >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Na= tive Method) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(Nat= iveMethodAccessorImpl.java:57) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke= (DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606= ) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.commons.daemon.support.DaemonLoader.= start(DaemonLoader.java:243) >>>>>>>>>>>>>>>>>>>>>>>>>>> Caused by: java.lang.UnsatisfiedLinkError: Can'= t >>>>>>>>>>>>>>>>>>>>>>>>>>> obtain updateLastError method for class com.sun= .jna.Native >>>>>>>>>>>>>>>>>>>>>>>>>>> at com.sun.jna.Native.initIDs(Native Method) >>>>>>>>>>>>>>>>>>>>>>>>>>> at com.sun.jna.Native.(Native.java:139) >>>>>>>>>>>>>>>>>>>>>>>>>>> at org.libvirt.jna.Libvirt.(Unknown >>>>>>>>>>>>>>>>>>>>>>>>>>> Source) >>>>>>>>>>>>>>>>>>>>>>>>>>> at org.libvirt.Library.(Unknown Source= ) >>>>>>>>>>>>>>>>>>>>>>>>>>> at org.libvirt.Connect.(Unknown Source) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnec= tion.getConnection(LibvirtConnection.java:44) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtConnec= tion.getConnection(LibvirtConnection.java:37) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComput= ingResource.configure(LibvirtComputingResource.java:733) >>>>>>>>>>>>>>>>>>>>>>>>>>> at com.cloud.agent.Agent.(Agent.java:161) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.agent.AgentShell.launchAgent(AgentShe= ll.java:415) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.agent.AgentShell.launchAgentFromClass= Info(AgentShell.java:370) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.agent.AgentShell.launchAgent(AgentShe= ll.java:351) >>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>> com.cloud.agent.AgentShell.start(AgentShell.jav= a:448) >>>>>>>>>>>>>>>>>>>>>>>>>>> ... 5 more >>>>>>>>>>>>>>>>>>>>>>>>>>> Cannot start daemon >>>>>>>>>>>>>>>>>>>>>>>>>>> Service exit with a return value of 5 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 7, 2013 at 2:31 PM, Mike Tutkowski = < >>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sure, that's a good plan. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll get to it. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 7, 2013 at 2:29 PM, Marcus Sorense= n >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know you mentioned you might need some mino= r >>>>>>>>>>>>>>>>>>>>>>>>>>>>> changes to it, as well as other minor changes= just for master (attach >>>>>>>>>>>>>>>>>>>>>>>>>>>>> volume switched to pool vs adapter or somethi= ng). My hope was that you >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be able to send an update that works fo= r your plugin on master, I'll >>>>>>>>>>>>>>>>>>>>>>>>>>>>> test against existing libvirtd storage and ap= ply it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Oct 7, 2013 1:49 PM, "Mike Tutkowski" < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is an automatically generated >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e-mail. To reply, visit: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://reviews.apache.org/r/14381/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This looks reasonable to me, Marcus. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When do you think you might start the proces= s of getting this into master? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Mike Tutkowski >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On September 30th, 2013, 5:14 p.m. UTC, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Marcus Sorensen wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Review request for cloudstack, edison su >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and Mike Tutkowski. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> By Marcus Sorensen. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Updated Sept. 30, 2013, 5:14 p.m.* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Repository: * cloudstack-git >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Description >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> With custom storage plugins comes the need t= o prep the KVM host prior to utilizing the disks. e.g. an iscsi initiator n= eeds to log into the target and scan for the lun before it can be used on t= he host. This patch is an example I developed against 4.2, minor changes ma= y be necessary to apply to master, but I want to share with others who are = working on storage so they can ensure it works for them. Please tweak as yo= u see fit. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MigrateCommand: pass vmTO object so we can s= ee which disks/storage pool types belong to the vm when migrating a VM. Thi= s facilitates being able to call disconnectPhysicalDisksViaVmSpec >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VirtualMachineManagerImpl: pass VirtualMachi= neTO when migrating so that we can see which disks belong to the VM and wha= t storage pools/adaptors should be used >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> LibvirtComputingResource: add calls KVMStora= gePoolManager's connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaV= mSpec calls where appropriate (when starting a vm, migrating a vm). Ensure = that we create 'raw' format XML disk definitions when the storage format is= RAW. Move cleanupDisk logic to storage adaptors so that each adaptor type = can clean up its disks in is own way. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KVMStoragePoolManager: add connectPhysicalD= isk, disconnectPhysicalDisk, connectPhysicalDiskViaVmSpec, disconnectPhysic= alDiskViaVmSpec, disconnectPhysicalDiskByPath. These all call the specific = StorageAdaptor's connectPhysicalDisk, disconnectPhysicalDisk, or disconnect= PhysicalDiskByPath calls. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KVMStorageProcessor: Call connectPhysicalDis= k/disconnectPhysicalDisk on the storage adaptor. Whether or not this is imp= lemented is up to the storage adaptor. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> LibvirtStorageAdaptor: implement dummy conne= ctPhysicalDisk/disconnectPhysicalDisk, move cleanupDisk logic from LibvirtC= omputingResource to disconnectPhysicalDiskByPath >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StorageAdaptor: define connectPhysicalDisk/d= isconnectPhysicalDisk/disconnectPhysicalDiskByPath in the interface >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Testing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Basic testing with my storage adaptor >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Diffs >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - core/src/com/cloud/agent/api/MigrateCom= mand.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (5042b8c) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - plugins/hypervisors/kvm/src/com/cloud/h= ypervisor/kvm/resource/LibvirtComputingResource.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3ee811f) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - plugins/hypervisors/kvm/src/com/cloud/h= ypervisor/kvm/storage/KVMStoragePoolManager.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (e09c9ba) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - plugins/hypervisors/kvm/src/com/cloud/h= ypervisor/kvm/storage/KVMStorageProcessor.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (c69f9b0) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - plugins/hypervisors/kvm/src/com/cloud/h= ypervisor/kvm/storage/LibvirtStorageAdaptor.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (123a9f1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - plugins/hypervisors/kvm/src/com/cloud/h= ypervisor/kvm/storage/StorageAdaptor.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (4956d8d) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - server/src/com/cloud/vm/VirtualMachineM= anagerImpl.java >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (d46bbb0) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> View Diff >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>>>> *=99* >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>> *=99* >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>> *=99* >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>> *=99* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Mike Tutkowski* >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>> e: mike.tutkowski@solidfire.com >>>>>>>>> o: 303.746.7302 >>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>> *=99* >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Mike Tutkowski* >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> e: mike.tutkowski@solidfire.com >>>>>>> o: 303.746.7302 >>>>>>> Advancing the way the world uses the cloud >>>>>>> *=99* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Mike Tutkowski* >>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> e: mike.tutkowski@solidfire.com >>>>>> o: 303.746.7302 >>>>>> Advancing the way the world uses the cloud >>>>>> *=99* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkowski@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the cloud >>>>> *=99* >>>>> >>>> >>>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkowski@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> *=99* >>> >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkowski@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> *=99* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkowski@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > *=99* > --=20 *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkowski@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud *=99* --089e01227ca495525004e86d6d08--