cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: Managed storage with KVM
Date Mon, 23 Sep 2013 22:54:45 GMT
maybe cloudstack-agent.out

On Mon, Sep 23, 2013 at 4:44 PM, Mike Tutkowski
<mike.tutkowski@solidfire.com> wrote:
> OK, so, nothing is screaming out in the logs. I did notice the following:
>
> From setup.log:
>
> DEBUG:root:execute:apparmor_status |grep libvirt
>
> DEBUG:root:Failed to execute:
>
>
> DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
>
> DEBUG:root:Failed to execute: * could not access PID file for
> cloudstack-agent
>
>
> This is the final line in this log file:
>
> DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start
>
>
> This is from agent.log:
>
> 2013-09-23 15:30:55,549 DEBUG [cloud.agent.AgentShell] (main:null) Checking
> to see if agent.pid exists.
>
> 2013-09-23 15:30:55,655 DEBUG [cloud.utils.ProcessUtil] (main:null)
> Executing: bash -c echo $PPID
>
> 2013-09-23 15:30:55,742 DEBUG [cloud.utils.ProcessUtil] (main:null)
> Execution is successful.
>
> 2013-09-23 15:30:56,000 INFO  [cloud.agent.Agent] (main:null) id is
>
> 2013-09-23 15:30:56,000 DEBUG [cloud.resource.ServerResourceBase]
> (main:null) Retrieving network interface: cloudbr0
>
> 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase]
> (main:null) Retrieving network interface: cloudbr0
>
> 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase]
> (main:null) Retrieving network interface: null
>
> 2013-09-23 15:30:56,017 DEBUG [cloud.resource.ServerResourceBase]
> (main:null) Retrieving network interface: null
>
>
> The following kinds of lines are repeated for a bunch of different .sh
> files. I think they often end up being found here:
> /usr/share/cloudstack-common/scripts/network/domr, so this is probably not
> an issue.
>
>
> 2013-09-23 15:30:56,111 DEBUG [utils.script.Script] (main:null) Looking for
> call_firewall.sh in the classpath
>
> 2013-09-23 15:30:56,112 DEBUG [utils.script.Script] (main:null) System
> resource: null
>
> 2013-09-23 15:30:56,113 DEBUG [utils.script.Script] (main:null) Classpath
> resource: null
>
> 2013-09-23 15:30:56,123 DEBUG [utils.script.Script] (main:null) Looking for
> call_firewall.sh
>
>
> Is there a log file for the Java code that I could write stuff out to and
> see how far we get?
>
>
> On Mon, Sep 23, 2013 at 3:17 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
>> Thanks, Marcus
>>
>> I've been developing on Windows for most of my time, so a bunch of these
>> Linux-type commands are new to me and I don't always interpret the output
>> correctly. Getting there. :)
>>
>>
>> On Mon, Sep 23, 2013 at 2:37 PM, Marcus Sorensen <shadowsor@gmail.com>wrote:
>>
>>> Nope, not running. That's just your grep process. It would look like:
>>>
>>> root     24429 24428  1 14:25 ?        00:00:08 jsvc.exec -cp
>>>
>>> /usr/share/java/commons-daemon.jar:/usr/share/java/jna.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjrt-1.7.
>>>
>>> Your agent log should tell you why it failed to start if you set it in
>>> debug and try to start... or maybe cloudstack-agent.out if it doesn't
>>> get far enough (say it's missing a class or something and can't
>>> start).
>>>
>>> On Mon, Sep 23, 2013 at 2:33 PM, Mike Tutkowski
>>> <mike.tutkowski@solidfire.com> wrote:
>>> > Looks like it's running, though:
>>> >
>>> > mtutkowski@ubuntu:~$ ps -ef | grep jsvc
>>> > 1000      7097  7013  0 14:32 pts/1    00:00:00 grep --color=auto jsvc
>>> >
>>> >
>>> >
>>> > On Mon, Sep 23, 2013 at 2:31 PM, Mike Tutkowski <
>>> > mike.tutkowski@solidfire.com> wrote:
>>> >
>>> >> Hey Marcus,
>>> >>
>>> >> Maybe you could give me a better idea of what the "flow" is when
>>> adding a
>>> >> KVM host.
>>> >>
>>> >> It looks like we SSH into the potential KVM host and execute a startup
>>> >> script (giving it necessary info about the cloud and the management
>>> server
>>> >> it should talk to).
>>> >>
>>> >> After this, is the Java VM started?
>>> >>
>>> >> After a reboot, I assume the JVM is started automatically?
>>> >>
>>> >> How do you debug your KVM-side Java code?
>>> >>
>>> >> Been looking through the logs and nothing obvious sticks out. I will
>>> have
>>> >> another look.
>>> >>
>>> >> Thanks
>>> >>
>>> >>
>>> >> On Mon, Sep 23, 2013 at 2:15 PM, Mike Tutkowski <
>>> >> mike.tutkowski@solidfire.com> wrote:
>>> >>
>>> >>> Hey Marcus,
>>> >>>
>>> >>> I've been investigating my issue with not being able to add a KVM
>>> host to
>>> >>> CS.
>>> >>>
>>> >>> For what it's worth, this comes back successful:
>>> >>>
>>> >>> SSHCmdHelper.sshExecuteCmd(sshConnection, "cloudstack-setup-agent " +
>>> >>> parameters, 3);
>>> >>>
>>> >>> This is what the command looks like:
>>> >>>
>>> >>> cloudstack-setup-agent  -m 192.168.233.1 -z 1 -p 1 -c 1 -g
>>> >>> 6b4aa1c2-2ac9-3c60-aabe-704aed40c684 -a --pubNic=cloudbr0
>>> --prvNic=cloudbr0
>>> >>> --guestNic=cloudbr0
>>> >>>
>>> >>> The problem is this method in LibvirtServerDiscoverer never finds a
>>> >>> matching host in the DB:
>>> >>>
>>> >>> waitForHostConnect(long dcId, long podId, long clusterId, String guid)
>>> >>>
>>> >>> I assume once the KVM host is up and running that it's supposed to
>>> call
>>> >>> into the CS MS so the DB can be updated as such?
>>> >>>
>>> >>> If so, the problem must be on the KVM side.
>>> >>>
>>> >>> I did run this again (from the KVM host) to see if the connection was
>>> in
>>> >>> place:
>>> >>>
>>> >>> mtutkowski@ubuntu:~$ telnet 192.168.233.1 8250
>>> >>>
>>> >>> Trying 192.168.233.1...
>>> >>>
>>> >>> Connected to 192.168.233.1.
>>> >>>
>>> >>> Escape character is '^]'.
>>> >>> So that looks good.
>>> >>>
>>> >>> I turned on more info in the debug log, but nothing obvious jumps out
>>> as
>>> >>> of yet.
>>> >>>
>>> >>> If you have any thoughts on this, please shoot them my way. :)
>>> >>>
>>> >>> Thanks!
>>> >>>
>>> >>>
>>> >>> On Sun, Sep 22, 2013 at 12:11 AM, Mike Tutkowski <
>>> >>> mike.tutkowski@solidfire.com> wrote:
>>> >>>
>>> >>>> First step is for me to get this working for KVM, though. :)
>>> >>>>
>>> >>>> Once I do that, I can perhaps make modifications to the storage
>>> >>>> framework and hypervisor plug-ins to refactor the logic and such.
>>> >>>>
>>> >>>>
>>> >>>> On Sun, Sep 22, 2013 at 12:09 AM, Mike Tutkowski <
>>> >>>> mike.tutkowski@solidfire.com> wrote:
>>> >>>>
>>> >>>>> Same would work for KVM.
>>> >>>>>
>>> >>>>> If CreateCommand and DestroyCommand were called at the appropriate
>>> >>>>> times by the storage framework, I could move my connect and
>>> disconnect
>>> >>>>> logic out of the attach/detach logic.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sun, Sep 22, 2013 at 12:08 AM, Mike Tutkowski <
>>> >>>>> mike.tutkowski@solidfire.com> wrote:
>>> >>>>>
>>> >>>>>> Conversely, if the storage framework called the DestroyCommand for
>>> >>>>>> managed storage after the DetachCommand, then I could have had my
>>> remove
>>> >>>>>> SR/datastore logic placed in the DestroyCommand handling rather
>>> than in the
>>> >>>>>> DetachCommand handling.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Sun, Sep 22, 2013 at 12:06 AM, Mike Tutkowski <
>>> >>>>>> mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>
>>> >>>>>>> Edison's plug-in calls the CreateCommand. Mine does not.
>>> >>>>>>>
>>> >>>>>>> The initial approach that was discussed during 4.2 was for me to
>>> >>>>>>> modify the attach/detach logic only in the XenServer and VMware
>>> hypervisor
>>> >>>>>>> plug-ins.
>>> >>>>>>>
>>> >>>>>>> Now that I think about it more, though, I kind of would have
>>> liked to
>>> >>>>>>> have the storage framework send a CreateCommand to the hypervisor
>>> before
>>> >>>>>>> sending the AttachCommand if the storage in question was managed.
>>> >>>>>>>
>>> >>>>>>> Then I could have created my SR/datastore in the CreateCommand and
>>> >>>>>>> the AttachCommand would have had the SR/datastore that it was
>>> always
>>> >>>>>>> expecting (and I wouldn't have had to create the SR/datastore in
>>> the
>>> >>>>>>> AttachCommand).
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Sat, Sep 21, 2013 at 11:56 PM, Marcus Sorensen <
>>> >>>>>>> shadowsor@gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>>> Yeah, I think it probably is as well, but I figured you'd be in a
>>> >>>>>>>> better position to tell.
>>> >>>>>>>>
>>> >>>>>>>> I see that copyAsync is unsupported in your current 4.2 driver,
>>> does
>>> >>>>>>>> that mean that there's no template support? Or is it some other
>>> call
>>> >>>>>>>> that does templating now? I'm still getting up to speed on all
>>> of the
>>> >>>>>>>> 4.2 changes. I was just looking at CreateCommand in
>>> >>>>>>>> LibvirtComputingResource, since that's the only place
>>> >>>>>>>> createPhysicalDisk is called, and it occurred to me that
>>> >>>>>>>> CreateCommand
>>> >>>>>>>> might be skipped altogether when utilizing storage plugins.
>>> >>>>>>>>
>>> >>>>>>>> On Sat, Sep 21, 2013 at 11:38 PM, Mike Tutkowski
>>> >>>>>>>> <mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>>> > That's an interesting comment, Marcus.
>>> >>>>>>>> >
>>> >>>>>>>> > It was my intent that it should work with any CloudStack
>>> "managed"
>>> >>>>>>>> storage
>>> >>>>>>>> > that uses an iSCSI target. Even though I'm using CHAP, I wrote
>>> the
>>> >>>>>>>> code so
>>> >>>>>>>> > CHAP didn't have to be used.
>>> >>>>>>>> >
>>> >>>>>>>> > As I'm doing my testing, I can try to think about whether it is
>>> >>>>>>>> generic
>>> >>>>>>>> > enough to keep those names or not.
>>> >>>>>>>> >
>>> >>>>>>>> > My expectation is that it is generic enough.
>>> >>>>>>>> >
>>> >>>>>>>> >
>>> >>>>>>>> > On Sat, Sep 21, 2013 at 11:32 PM, Marcus Sorensen <
>>> >>>>>>>> shadowsor@gmail.com>wrote:
>>> >>>>>>>> >
>>> >>>>>>>> >> I added a comment to your diff. In general I think it looks
>>> good,
>>> >>>>>>>> >> though I obviously can't vouch for whether or not it will
>>> work.
>>> >>>>>>>> One
>>> >>>>>>>> >> thing I do have reservations about is the adaptor/pool
>>> naming. If
>>> >>>>>>>> you
>>> >>>>>>>> >> think the code is generic enough that it will work for anyone
>>> who
>>> >>>>>>>> does
>>> >>>>>>>> >> an iscsi LUN-per-volume plugin, then it's OK, but if there's
>>> >>>>>>>> anything
>>> >>>>>>>> >> about it that's specific to YOUR iscsi target or how it likes
>>> to
>>> >>>>>>>> be
>>> >>>>>>>> >> treated then I'd say that they should be named something less
>>> >>>>>>>> generic
>>> >>>>>>>> >> than iScsiAdmStorage.
>>> >>>>>>>> >>
>>> >>>>>>>> >> On Sat, Sep 21, 2013 at 7:23 PM, Mike Tutkowski
>>> >>>>>>>> >> <mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>>> >> > Great - thanks!
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > Just to give you an overview of what my code does (for when
>>> you
>>> >>>>>>>> get a
>>> >>>>>>>> >> > chance to review it):
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > SolidFireHostListener is registered in
>>> >>>>>>>> SolidfirePrimaryDataStoreProvider.
>>> >>>>>>>> >> > Its hostConnect method is invoked when a host connects with
>>> the
>>> >>>>>>>> CS MS. If
>>> >>>>>>>> >> > the host is running KVM, the listener sends a
>>> >>>>>>>> ModifyStoragePoolCommand to
>>> >>>>>>>> >> > the host. This logic was based off of DefaultHostListener.
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > The handling of ModifyStoragePoolCommand is unchanged. It
>>> >>>>>>>> invokes
>>> >>>>>>>> >> > createStoragePool on the KVMStoragePoolManager. The
>>> >>>>>>>> KVMStoragePoolManager
>>> >>>>>>>> >> > asks for an adaptor and finds my new one:
>>> >>>>>>>> iScsiAdmStorageAdaptor (which
>>> >>>>>>>> >> was
>>> >>>>>>>> >> > registered in the constructor for KVMStoragePoolManager
>>> under
>>> >>>>>>>> the key of
>>> >>>>>>>> >> > StoragePoolType.Iscsi.toString()).
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > iScsiAdmStorageAdaptor.createStoragePool just makes an
>>> instance
>>> >>>>>>>> of
>>> >>>>>>>> >> > iScsiAdmStoragePool, adds it to a map, and returns the
>>> pointer
>>> >>>>>>>> to the
>>> >>>>>>>> >> > iScsiAdmStoragePool object. The key of the map is the UUID
>>> of
>>> >>>>>>>> the storage
>>> >>>>>>>> >> > pool.
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > When a volume is attached, createPhysicalDisk is invoked for
>>> >>>>>>>> managed
>>> >>>>>>>> >> > storage rather than getPhysicalDisk. createPhysicalDisk uses
>>> >>>>>>>> iscsiadm to
>>> >>>>>>>> >> > establish the iSCSI connection to the volume on the SAN and
>>> a
>>> >>>>>>>> >> > KVMPhysicalDisk is returned to be used in the attach logic
>>> that
>>> >>>>>>>> follows.
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > When a volume is detached, getPhysicalDisk is invoked with
>>> the
>>> >>>>>>>> IQN of the
>>> >>>>>>>> >> > volume if the storage pool in question is managed storage.
>>> >>>>>>>> Otherwise, the
>>> >>>>>>>> >> > normal vol.getPath() is used.
>>> >>>>>>>> iScsiAdmStorageAdaptor.getPhysicalDisk just
>>> >>>>>>>> >> > returns a new instance of KVMPhysicalDisk to be used in the
>>> >>>>>>>> detach logic.
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > Once the volume has been detached,
>>> >>>>>>>> iScsiAdmStoragePool.deletePhysicalDisk
>>> >>>>>>>> >> > is invoked if the storage pool is managed.
>>> deletePhysicalDisk
>>> >>>>>>>> removes the
>>> >>>>>>>> >> > iSCSI connection to the volume using iscsiadm.
>>> >>>>>>>> >> >
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > On Sat, Sep 21, 2013 at 5:46 PM, Marcus Sorensen <
>>> >>>>>>>> shadowsor@gmail.com
>>> >>>>>>>> >> >wrote:
>>> >>>>>>>> >> >
>>> >>>>>>>> >> >> Its the log4j properties file in /etc/cloudstack/agent
>>> change
>>> >>>>>>>> all INFO
>>> >>>>>>>> >> to
>>> >>>>>>>> >> >> DEBUG.  I imagine the agent just isn't starting, you can
>>> tail
>>> >>>>>>>> the log
>>> >>>>>>>> >> when
>>> >>>>>>>> >> >> you try to start the service, or maybe it will spit
>>> something
>>> >>>>>>>> out into
>>> >>>>>>>> >> one
>>> >>>>>>>> >> >> of the other files in /var/log/cloudstack/agent
>>> >>>>>>>> >> >> On Sep 21, 2013 5:19 PM, "Mike Tutkowski" <
>>> >>>>>>>> mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >
>>> >>>>>>>> >> >> wrote:
>>> >>>>>>>> >> >>
>>> >>>>>>>> >> >> > This is how I've been trying to query for the status of
>>> the
>>> >>>>>>>> service (I
>>> >>>>>>>> >> >> > assume it could be started this way, as well, by changing
>>> >>>>>>>> "status" to
>>> >>>>>>>> >> >> > "start" or "restart"?):
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo
>>> >>>>>>>> /usr/sbin/service
>>> >>>>>>>> >> >> > cloudstack-agent status
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > I get this back:
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > Failed to execute: * could not access PID file for
>>> >>>>>>>> cloudstack-agent
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > I've made a bunch of code changes recently, though, so I
>>> >>>>>>>> think I'm
>>> >>>>>>>> >> going
>>> >>>>>>>> >> >> to
>>> >>>>>>>> >> >> > rebuild and redeploy everything.
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > The debug info sounds helpful. Where can I set
>>> enable.debug?
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > Thanks, Marcus!
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > On Sat, Sep 21, 2013 at 4:11 PM, Marcus Sorensen <
>>> >>>>>>>> shadowsor@gmail.com
>>> >>>>>>>> >> >> > >wrote:
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > > OK, will check it out in the next few days. As
>>> mentioned,
>>> >>>>>>>> you can
>>> >>>>>>>> >> set
>>> >>>>>>>> >> >> up
>>> >>>>>>>> >> >> > > your Ubuntu vm as the management server as well if all
>>> >>>>>>>> else fails.
>>> >>>>>>>> >>  If
>>> >>>>>>>> >> >> > you
>>> >>>>>>>> >> >> > > can get to the mgmt server on 8250 from the KVM host,
>>> then
>>> >>>>>>>> you need
>>> >>>>>>>> >> to
>>> >>>>>>>> >> >> > > enable.debug on the agent. It won't run without
>>> >>>>>>>> complaining loudly
>>> >>>>>>>> >> if
>>> >>>>>>>> >> >> it
>>> >>>>>>>> >> >> > > can't get to the mgmt server, and I didn't see that in
>>> >>>>>>>> your agent
>>> >>>>>>>> >> log,
>>> >>>>>>>> >> >> so
>>> >>>>>>>> >> >> > > perhaps its not running. I assume you know how to
>>> >>>>>>>> stop/start the
>>> >>>>>>>> >> agent
>>> >>>>>>>> >> >> on
>>> >>>>>>>> >> >> > > KVM via 'service cloud stacks agent'.
>>> >>>>>>>> >> >> > > On Sep 21, 2013 3:02 PM, "Mike Tutkowski" <
>>> >>>>>>>> >> >> mike.tutkowski@solidfire.com>
>>> >>>>>>>> >> >> > > wrote:
>>> >>>>>>>> >> >> > >
>>> >>>>>>>> >> >> > > > Hey Marcus,
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > I haven't yet been able to test my new code, but I
>>> >>>>>>>> thought you
>>> >>>>>>>> >> would
>>> >>>>>>>> >> >> > be a
>>> >>>>>>>> >> >> > > > good person to ask to review it:
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > >
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >>
>>> >>>>>>>> >>
>>> >>>>>>>>
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/ea74b312a8a36801994500407fd54f0cdda55e37
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > All it is supposed to do is attach and detach a data
>>> >>>>>>>> disk (that
>>> >>>>>>>> >> has
>>> >>>>>>>> >> >> > > > guaranteed IOPS) with KVM as the hypervisor. The data
>>> >>>>>>>> disk
>>> >>>>>>>> >> happens to
>>> >>>>>>>> >> >> > be
>>> >>>>>>>> >> >> > > > from SolidFire-backed storage - where we have a 1:1
>>> >>>>>>>> mapping
>>> >>>>>>>> >> between a
>>> >>>>>>>> >> >> > > > CloudStack volume and a data disk.
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > There is no support for hypervisor snapshots or stuff
>>> >>>>>>>> like that
>>> >>>>>>>> >> >> > (likely a
>>> >>>>>>>> >> >> > > > future release)...just attaching and detaching a data
>>> >>>>>>>> disk in 4.3.
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > Thanks!
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > On Fri, Sep 20, 2013 at 11:05 PM, Mike Tutkowski <
>>> >>>>>>>> >> >> > > > mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > > When I re-deployed the DEBs, I didn't remove
>>> >>>>>>>> cloudstack-agent
>>> >>>>>>>> >> >> first.
>>> >>>>>>>> >> >> > > > Would
>>> >>>>>>>> >> >> > > > > that be a problem? I just did a sudo apt-get
>>> install
>>> >>>>>>>> >> >> > cloudstack-agent.
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > > > On Fri, Sep 20, 2013 at 11:03 PM, Mike Tutkowski <
>>> >>>>>>>> >> >> > > > > mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > > >> I get the same error running the command manually:
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >> mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo
>>> >>>>>>>> >> /usr/sbin/service
>>> >>>>>>>> >> >> > > > >> cloudstack-agent status
>>> >>>>>>>> >> >> > > > >>  * could not access PID file for cloudstack-agent
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >> On Fri, Sep 20, 2013 at 11:00 PM, Mike Tutkowski <
>>> >>>>>>>> >> >> > > > >> mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >>> agent.log looks OK to me:
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,010 INFO
>>> >>>>>>>>  [cloud.agent.AgentShell]
>>> >>>>>>>> >> >> (main:null)
>>> >>>>>>>> >> >> > > > Agent
>>> >>>>>>>> >> >> > > > >>> started
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,011 INFO
>>> >>>>>>>>  [cloud.agent.AgentShell]
>>> >>>>>>>> >> >> (main:null)
>>> >>>>>>>> >> >> > > > >>> Implementation Version is 4.3.0-SNAPSHOT
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,015 INFO
>>> >>>>>>>>  [cloud.agent.AgentShell]
>>> >>>>>>>> >> >> (main:null)
>>> >>>>>>>> >> >> > > > >>> agent.properties found at
>>> >>>>>>>> >> /etc/cloudstack/agent/agent.properties
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,023 INFO
>>> >>>>>>>>  [cloud.agent.AgentShell]
>>> >>>>>>>> >> >> (main:null)
>>> >>>>>>>> >> >> > > > >>> Defaulting to using properties file for storage
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,024 INFO
>>> >>>>>>>>  [cloud.agent.AgentShell]
>>> >>>>>>>> >> >> (main:null)
>>> >>>>>>>> >> >> > > > >>> Defaulting to the constant time backoff algorithm
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,029 INFO
>>>  [cloud.utils.LogUtils]
>>> >>>>>>>> >> (main:null)
>>> >>>>>>>> >> >> > > log4j
>>> >>>>>>>> >> >> > > > >>> configuration found at
>>> >>>>>>>> /etc/cloudstack/agent/log4j-cloud.xml
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,163 INFO  [cloud.agent.Agent]
>>> >>>>>>>> (main:null)
>>> >>>>>>>> >> id
>>> >>>>>>>> >> >> > is 3
>>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,197 INFO
>>> >>>>>>>> >> >> > > > >>>  [resource.virtualnetwork.VirtualRoutingResource]
>>> >>>>>>>> (main:null)
>>> >>>>>>>> >> >> > > > >>> VirtualRoutingResource _scriptDir to use:
>>> >>>>>>>> >> >> scripts/network/domr/kvm
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>> However, I wasn't aware that setup.log was
>>> >>>>>>>> important. This
>>> >>>>>>>> >> seems
>>> >>>>>>>> >> >> to
>>> >>>>>>>> >> >> > > be
>>> >>>>>>>> >> >> > > > a
>>> >>>>>>>> >> >> > > > >>> problem, but I'm not sure what it might indicate:
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service
>>> >>>>>>>> cloudstack-agent
>>> >>>>>>>> >> status
>>> >>>>>>>> >> >> > > > >>> DEBUG:root:Failed to execute: * could not access
>>> PID
>>> >>>>>>>> file for
>>> >>>>>>>> >> >> > > > >>> cloudstack-agent
>>> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service
>>> >>>>>>>> cloudstack-agent
>>> >>>>>>>> >> start
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>> On Fri, Sep 20, 2013 at 10:53 PM, Marcus
>>> Sorensen <
>>> >>>>>>>> >> >> > > shadowsor@gmail.com
>>> >>>>>>>> >> >> > > > >wrote:
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>>> Sorry, I saw that in the log, I thought it was
>>> the
>>> >>>>>>>> agent log
>>> >>>>>>>> >> for
>>> >>>>>>>> >> >> > > some
>>> >>>>>>>> >> >> > > > >>>> reason. Is the agent started? That might be the
>>> >>>>>>>> place to
>>> >>>>>>>> >> look.
>>> >>>>>>>> >> >> > There
>>> >>>>>>>> >> >> > > > is
>>> >>>>>>>> >> >> > > > >>>> an
>>> >>>>>>>> >> >> > > > >>>> agent log for the agent and one for the setup
>>> when
>>> >>>>>>>> it adds
>>> >>>>>>>> >> the
>>> >>>>>>>> >> >> > host,
>>> >>>>>>>> >> >> > > > >>>> both
>>> >>>>>>>> >> >> > > > >>>> in /var/log
>>> >>>>>>>> >> >> > > > >>>> On Sep 20, 2013 10:42 PM, "Mike Tutkowski" <
>>> >>>>>>>> >> >> > > > >>>> mike.tutkowski@solidfire.com>
>>> >>>>>>>> >> >> > > > >>>> wrote:
>>> >>>>>>>> >> >> > > > >>>>
>>> >>>>>>>> >> >> > > > >>>> > Is it saying that the MS is at the IP address
>>> or
>>> >>>>>>>> the KVM
>>> >>>>>>>> >> host?
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> > The KVM host is at 192.168.233.10.
>>> >>>>>>>> >> >> > > > >>>> > The MS host is at 192.168.233.1.
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> > I see this for my host Global Settings
>>> parameter:
>>> >>>>>>>> >> >> > > > >>>> > hostThe ip address of management
>>> >>>>>>>> server192.168.233.1
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> > /etc/cloudstack/agent/agent.properties has a
>>> >>>>>>>> >> >> host=192.168.233.1
>>> >>>>>>>> >> >> > > > value.
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> > On Fri, Sep 20, 2013 at 10:21 PM, Marcus
>>> Sorensen
>>> >>>>>>>> <
>>> >>>>>>>> >> >> > > > >>>> shadowsor@gmail.com
>>> >>>>>>>> >> >> > > > >>>> > >wrote:
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> > > The log says your mgmt server is
>>> >>>>>>>> 192.168.233.10? But you
>>> >>>>>>>> >> >> tried
>>> >>>>>>>> >> >> > > to
>>> >>>>>>>> >> >> > > > >>>> telnet
>>> >>>>>>>> >> >> > > > >>>> > to
>>> >>>>>>>> >> >> > > > >>>> > > 192.168.233.1? It might be enough to change
>>> >>>>>>>> that in
>>> >>>>>>>> >> >> > > > >>>> > > /etc/cloudstack/agent/agent.properties, but
>>> you
>>> >>>>>>>> may want
>>> >>>>>>>> >> to
>>> >>>>>>>> >> >> > edit
>>> >>>>>>>> >> >> > > > the
>>> >>>>>>>> >> >> > > > >>>> > config
>>> >>>>>>>> >> >> > > > >>>> > > as well to tell it the real ms IP.
>>> >>>>>>>> >> >> > > > >>>> > > On Sep 20, 2013 10:12 PM, "Mike Tutkowski" <
>>> >>>>>>>> >> >> > > > >>>> mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > >>>> > >
>>> >>>>>>>> >> >> > > > >>>> > > wrote:
>>> >>>>>>>> >> >> > > > >>>> > >
>>> >>>>>>>> >> >> > > > >>>> > > > Here's what my /etc/network/interfaces
>>> file
>>> >>>>>>>> looks
>>> >>>>>>>> >> like, if
>>> >>>>>>>> >> >> > > that
>>> >>>>>>>> >> >> > > > >>>> is of
>>> >>>>>>>> >> >> > > > >>>> > > > interest (the 192.168.233.0 network is the
>>> >>>>>>>> NAT network
>>> >>>>>>>> >> >> > VMware
>>> >>>>>>>> >> >> > > > >>>> Fusion
>>> >>>>>>>> >> >> > > > >>>> > set
>>> >>>>>>>> >> >> > > > >>>> > > > up):
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > > auto lo
>>> >>>>>>>> >> >> > > > >>>> > > > iface lo inet loopback
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > > auto eth0
>>> >>>>>>>> >> >> > > > >>>> > > > iface eth0 inet manual
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > > auto cloudbr0
>>> >>>>>>>> >> >> > > > >>>> > > > iface cloudbr0 inet static
>>> >>>>>>>> >> >> > > > >>>> > > >     address 192.168.233.10
>>> >>>>>>>> >> >> > > > >>>> > > >     netmask 255.255.255.0
>>> >>>>>>>> >> >> > > > >>>> > > >     network 192.168.233.0
>>> >>>>>>>> >> >> > > > >>>> > > >     broadcast 192.168.233.255
>>> >>>>>>>> >> >> > > > >>>> > > >     dns-nameservers 8.8.8.8
>>> >>>>>>>> >> >> > > > >>>> > > >     bridge_ports eth0
>>> >>>>>>>> >> >> > > > >>>> > > >     bridge_fd 5
>>> >>>>>>>> >> >> > > > >>>> > > >     bridge_stp off
>>> >>>>>>>> >> >> > > > >>>> > > >     bridge_maxwait 1
>>> >>>>>>>> >> >> > > > >>>> > > >     post-up route add default gw
>>> >>>>>>>> 192.168.233.2 metric 1
>>> >>>>>>>> >> >> > > > >>>> > > >     pre-down route del default gw
>>> >>>>>>>> 192.168.233.2
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > > On Fri, Sep 20, 2013 at 10:08 PM, Mike
>>> >>>>>>>> Tutkowski <
>>> >>>>>>>> >> >> > > > >>>> > > > mike.tutkowski@solidfire.com> wrote:
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > You appear to be correct. This is from
>>> the
>>> >>>>>>>> MS log
>>> >>>>>>>> >> >> (below).
>>> >>>>>>>> >> >> > > > >>>> Discovery
>>> >>>>>>>> >> >> > > > >>>> > > > timed
>>> >>>>>>>> >> >> > > > >>>> > > > > out.
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > I'm not sure why this would be. My
>>> network
>>> >>>>>>>> settings
>>> >>>>>>>> >> >> > > shouldn't
>>> >>>>>>>> >> >> > > > >>>> have
>>> >>>>>>>> >> >> > > > >>>> > > > changed
>>> >>>>>>>> >> >> > > > >>>> > > > > since the last time I tried this.
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > I am able to ping the KVM host from the
>>> MS
>>> >>>>>>>> host and
>>> >>>>>>>> >> vice
>>> >>>>>>>> >> >> > > > versa.
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > I'm even able to manually kick off a VM
>>> on
>>> >>>>>>>> the KVM
>>> >>>>>>>> >> host
>>> >>>>>>>> >> >> > and
>>> >>>>>>>> >> >> > > > >>>> ping from
>>> >>>>>>>> >> >> > > > >>>> > > it
>>> >>>>>>>> >> >> > > > >>>> > > > > to the MS host.
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > I am using NAT from my Mac OS X host
>>> (also
>>> >>>>>>>> running
>>> >>>>>>>> >> the
>>> >>>>>>>> >> >> CS
>>> >>>>>>>> >> >> > > MS)
>>> >>>>>>>> >> >> > > > >>>> to the
>>> >>>>>>>> >> >> > > > >>>> > VM
>>> >>>>>>>> >> >> > > > >>>> > > > > running KVM (VMware Fusion).
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,141 DEBUG
>>> >>>>>>>> >> >> > > > >>>> [c.c.h.k.d.LibvirtServerDiscoverer]
>>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3
>>> :ctx-6b28dc48)
>>> >>>>>>>> Timeout,
>>> >>>>>>>> >> to
>>> >>>>>>>> >> >> > wait
>>> >>>>>>>> >> >> > > > for
>>> >>>>>>>> >> >> > > > >>>> the
>>> >>>>>>>> >> >> > > > >>>> > > host
>>> >>>>>>>> >> >> > > > >>>> > > > > connecting to mgt svr, assuming it is
>>> failed
>>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,144 WARN
>>> >>>>>>>> >> >>  [c.c.r.ResourceManagerImpl]
>>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3
>>> :ctx-6b28dc48)
>>> >>>>>>>> Unable to
>>> >>>>>>>> >> >> find
>>> >>>>>>>> >> >> > > the
>>> >>>>>>>> >> >> > > > >>>> server
>>> >>>>>>>> >> >> > > > >>>> > > > > resources at http://192.168.233.10
>>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,145 INFO
>>> >>>>>>>> >> >> >  [c.c.u.e.CSExceptionErrorCode]
>>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3
>>> :ctx-6b28dc48)
>>> >>>>>>>> Could not
>>> >>>>>>>> >> >> find
>>> >>>>>>>> >> >> > > > >>>> exception:
>>> >>>>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException
>>> in
>>> >>>>>>>> error code
>>> >>>>>>>> >> >> list
>>> >>>>>>>> >> >> > > for
>>> >>>>>>>> >> >> > > > >>>> > > exceptions
>>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,147 WARN
>>> >>>>>>>> >>  [o.a.c.a.c.a.h.AddHostCmd]
>>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3
>>> :ctx-6b28dc48)
>>> >>>>>>>> Exception:
>>> >>>>>>>> >> >> > > > >>>> > > > > com.cloud.exception.DiscoveryException:
>>> >>>>>>>> Unable to add
>>> >>>>>>>> >> >> the
>>> >>>>>>>> >> >> > > host
>>> >>>>>>>> >> >> > > > >>>> > > > > at
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > >
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>>
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > >
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >>
>>> >>>>>>>> >>
>>> >>>>>>>>
>>> com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:778)
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > I do seem to be able to telnet in from
>>> my
>>> >>>>>>>> KVM host to
>>> >>>>>>>> >> >> the
>>> >>>>>>>> >> >> > MS
>>> >>>>>>>> >> >> > > > >>>> host's
>>> >>>>>>>> >> >> > > > >>>> > > 8250
>>> >>>>>>>> >> >> > > > >>>> > > > > port:
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > > > mtutkowski@ubuntu:~$ telnet
>>> 192.168.233.1
>>> >>>>>>>> 8250
>>> >>>>>>>> >> >> > > > >>>> > > > > Trying 192.168.233.1...
>>> >>>>>>>> >> >> > > > >>>> > > > > Connected to 192.168.233.1.
>>> >>>>>>>> >> >> > > > >>>> > > > > Escape character is '^]'.
>>> >>>>>>>> >> >> > > > >>>> > > > >
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > > > --
>>> >>>>>>>> >> >> > > > >>>> > > > *Mike Tutkowski*
>>> >>>>>>>> >> >> > > > >>>> > > > *Senior CloudStack Developer, SolidFire
>>> Inc.*
>>> >>>>>>>> >> >> > > > >>>> > > > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > >>>> > > > o: 303.746.7302
>>> >>>>>>>> >> >> > > > >>>> > > > Advancing the way the world uses the
>>> >>>>>>>> >> >> > > > >>>> > > > cloud<
>>> >>>>>>>> >> http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> >> >> > > > >>>> > > > *™*
>>> >>>>>>>> >> >> > > > >>>> > > >
>>> >>>>>>>> >> >> > > > >>>> > >
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>> > --
>>> >>>>>>>> >> >> > > > >>>> > *Mike Tutkowski*
>>> >>>>>>>> >> >> > > > >>>> > *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> >> > > > >>>> > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > >>>> > o: 303.746.7302
>>> >>>>>>>> >> >> > > > >>>> > Advancing the way the world uses the
>>> >>>>>>>> >> >> > > > >>>> > cloud<
>>> >>>>>>>> http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> >> >> > > > >>>> > *™*
>>> >>>>>>>> >> >> > > > >>>> >
>>> >>>>>>>> >> >> > > > >>>>
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>> --
>>> >>>>>>>> >> >> > > > >>> *Mike Tutkowski*
>>> >>>>>>>> >> >> > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> >> > > > >>> e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > >>> o: 303.746.7302
>>> >>>>>>>> >> >> > > > >>> Advancing the way the world uses the cloud<
>>> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> >> >> > > > >>> *™*
>>> >>>>>>>> >> >> > > > >>>
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >> --
>>> >>>>>>>> >> >> > > > >> *Mike Tutkowski*
>>> >>>>>>>> >> >> > > > >> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> >> > > > >> e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > >> o: 303.746.7302
>>> >>>>>>>> >> >> > > > >> Advancing the way the world uses the cloud<
>>> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> >> >> > > > >> *™*
>>> >>>>>>>> >> >> > > > >>
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > > > --
>>> >>>>>>>> >> >> > > > > *Mike Tutkowski*
>>> >>>>>>>> >> >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> >> > > > > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > > o: 303.746.7302
>>> >>>>>>>> >> >> > > > > Advancing the way the world uses the cloud<
>>> >>>>>>>> >> >> > > > http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> >> >> > > > > *™*
>>> >>>>>>>> >> >> > > > >
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > > > --
>>> >>>>>>>> >> >> > > > *Mike Tutkowski*
>>> >>>>>>>> >> >> > > > *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> >> > > > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > > > o: 303.746.7302
>>> >>>>>>>> >> >> > > > Advancing the way the world uses the
>>> >>>>>>>> >> >> > > > cloud<
>>> http://solidfire.com/solution/overview/?video=play
>>> >>>>>>>> >
>>> >>>>>>>> >> >> > > > *™*
>>> >>>>>>>> >> >> > > >
>>> >>>>>>>> >> >> > >
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >> > --
>>> >>>>>>>> >> >> > *Mike Tutkowski*
>>> >>>>>>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> >> > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> >> > o: 303.746.7302
>>> >>>>>>>> >> >> > Advancing the way the world uses the
>>> >>>>>>>> >> >> > cloud<http://solidfire.com/solution/overview/?video=play
>>> >
>>> >>>>>>>> >> >> > *™*
>>> >>>>>>>> >> >> >
>>> >>>>>>>> >> >>
>>> >>>>>>>> >> >
>>> >>>>>>>> >> >
>>> >>>>>>>> >> >
>>> >>>>>>>> >> > --
>>> >>>>>>>> >> > *Mike Tutkowski*
>>> >>>>>>>> >> > *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> >> > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> >> > o: 303.746.7302
>>> >>>>>>>> >> > Advancing the way the world uses the
>>> >>>>>>>> >> > cloud<http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> >> > *™*
>>> >>>>>>>> >>
>>> >>>>>>>> >
>>> >>>>>>>> >
>>> >>>>>>>> >
>>> >>>>>>>> > --
>>> >>>>>>>> > *Mike Tutkowski*
>>> >>>>>>>> > *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>>> > e: mike.tutkowski@solidfire.com
>>> >>>>>>>> > o: 303.746.7302
>>> >>>>>>>> > Advancing the way the world uses the
>>> >>>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>>> > *™*
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> *Mike Tutkowski*
>>> >>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>>> e: mike.tutkowski@solidfire.com
>>> >>>>>>> o: 303.746.7302
>>> >>>>>>> Advancing the way the world uses the cloud<
>>> http://solidfire.com/solution/overview/?video=play>
>>> >>>>>>> *™*
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> *Mike Tutkowski*
>>> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>>> e: mike.tutkowski@solidfire.com
>>> >>>>>> o: 303.746.7302
>>> >>>>>> Advancing the way the world uses the cloud<
>>> http://solidfire.com/solution/overview/?video=play>
>>> >>>>>> *™*
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> *Mike Tutkowski*
>>> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>>> e: mike.tutkowski@solidfire.com
>>> >>>>> o: 303.746.7302
>>> >>>>> Advancing the way the world uses the cloud<
>>> http://solidfire.com/solution/overview/?video=play>
>>> >>>>> *™*
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> *Mike Tutkowski*
>>> >>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>>> e: mike.tutkowski@solidfire.com
>>> >>>> o: 303.746.7302
>>> >>>> Advancing the way the world uses the cloud<
>>> http://solidfire.com/solution/overview/?video=play>
>>> >>>> *™*
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> *Mike Tutkowski*
>>> >>> *Senior CloudStack Developer, SolidFire Inc.*
>>> >>> e: mike.tutkowski@solidfire.com
>>> >>> o: 303.746.7302
>>> >>> Advancing the way the world uses the cloud<
>>> http://solidfire.com/solution/overview/?video=play>
>>> >>> *™*
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> *Mike Tutkowski*
>>> >> *Senior CloudStack Developer, SolidFire Inc.*
>>> >> e: mike.tutkowski@solidfire.com
>>> >> o: 303.746.7302
>>> >> Advancing the way the world uses the cloud<
>>> http://solidfire.com/solution/overview/?video=play>
>>> >> *™*
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > *Mike Tutkowski*
>>> > *Senior CloudStack Developer, SolidFire Inc.*
>>> > e: mike.tutkowski@solidfire.com
>>> > o: 303.746.7302
>>> > Advancing the way the world uses the
>>> > cloud<http://solidfire.com/solution/overview/?video=play>
>>> > *™*
>>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>> *™*
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*

Mime
View raw message