cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Musayev, Ilya" <>
Subject RE: Introduction
Date Thu, 22 Aug 2013 03:41:47 GMT
Welcome back Darren!

Seems like we have a lot in common, I tend to leave "no's and not's" quite often! 

Since your experience was based on older CS 2.x system, I wonder how different (better or
worse) it is now? Are we closer to GoDaddy hybrid solution?

-----Original Message-----
From: Frank Zhang [] 
Sent: Wednesday, August 21, 2013 6:59 PM
Subject: RE: Introduction

Welcome back!

> -----Original Message-----
> From: Darren Shepherd []
> Sent: Wednesday, August 21, 2013 12:17 PM
> To:
> Subject: Introduction
>  All,
> I want to introduce myself to the Apache CloudStack community.  I've 
> have had a long love/hate relationship with CloudStack over the years, 
> but recently I've decided to fully dedicate myself to working on this platform (again).
> Just a little about me, well okay, this might be a little long.  I've 
> been a Linux user for a good amount of time.  I've ran Linux as my 
> primary OS for over 15 years.  Both personally and professionally, my 
> desktops/laptops have always ran Linux.  As soon as I discovered Linux I also fell in
love with virtualization.
> I've ran the gamut in terms of virtualization technologies.  Type 1, 
> Type 2, containers, etc.  I have a particular fondness for Xen though.  
> I've been a Xen user since the 2.x days.
> Couple years after I got into Linux I discovered programming.  After 
> so many failed "./configure && make" I figured I'd should probably 
> learn C/C++ to figure out why the heck nothing ever "make'd" right.  I 
> worked for awhile running a lab for developers that were writing stuff 
> in Java.  Again I learned Java to figure out why the heck their stuff 
> kept failing to build and run.  After I learned Java I took the 
> professional route of leaving the sysadmin stuff and started programming distributed
> As means to an end I developed a system that would take code builds, 
> deploy them to Xen thin provisioned VMs.  So a developer could go to a 
> portal, click some stuff and get a clean VM that optionally had a code 
> build on it.  We would then running automated integration tests against it.  This was
around 2006.
> About a year or 2 later I discovered EC2 and was like "holy crap, 
> that's exactly what I want."  Then I found eucalpytus and realized 
> there was a whole industry around this stuff.  IaaS is basically the 
> trifecta for me.  I get to combine my love of Linux, virtualization,
> and programming distributed system.   Basically since then I've been
> absolutely obsessed with IaaS.
> Around 2010 I got a job at Go Daddy specifically to build their public 
> Cloud product.  Before I even got there they had already decided to go 
> with vmops/  I'm pretty sure they were already when I started.
> At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java guys, 
> 2 sysengs) we built three generations of clouds.  The first generation 
> of cloud was basically CloudStack 2.2++.  It was CloudStack 2.2 with a 
> new UI, billing system, storage subsystem, and a tons of operational 
> tools.  That was the initial public launch of our Cloud.
> The second generation was "CloudStack like."  Basically we had a lot 
> of problems with CloudStack at the time.  First, it was just a 
> complete pain to operationalize.  Trying to hand off CloudStack to our 
> ops and support teams was practically impossible.  Basically our level 
> 1 and 2 support would be very efficient at escalating all issues to level 3 (my team).
> Typically the sysadmins would see something is stuck, they'd look at a 
> log (that made no sense to them at all), see a 100 line stack trace 
> and then they'd just toss the issue to the developers.  Also, their 
> was no visibility to the general health of CloudStack.  If you asked ops, "How's production
doing right now?"
> Their answer would be, "Well I can ping the UI and the java processes 
> are still running."  In developer terms that is kinda like saying, 
> "Well it compiles so it must work fine."
> Secondly, we had a hard time extending the platform.  We always seemed 
> to be a little ahead of the curve in how we wanted to implement the cloud.
> How we wanted to run storage and networking didn't match at all what 
> CloudStack was at that time.  So we embarked on improving it.  It was 
> just way too difficult.  Basically we felt like 70% of our dev time 
> was just fighting the framework.  So we made the choice to abandon the 
> platform.  We started rewriting the the entire platform.  We did this 
> piece meal over time.  We'd redo a subsystem, put that in production, 
> and then move onto the next subsystem.  This ended up being very 
> tricky because we had mixture of CloudStack and our own stuff.  We 
> also always tried to do no downtime deployments.  So we would almost 
> never lock people out of the UI and most deployments were during 
> business hours.  We basically finished most of this work around summer 
> last year.  I gave a talk at OSCON 2012 [1] regarding what we built.  
> A couple months later, solely based on business direction, we decided 
> to stop selling the public cloud (it really wasn't shutdown until 
> about May this year.  I'm proud to say it ran for almost 8 months with live customers,
some quite large, with no incident and no dedicated ops team).
> After we shutdown the public cloud, we turned our effort to building a 
> private cloud to run Go Daddy on top of it.  This marked the third 
> generation of the cloud.  At this point the IaaS system was 100% our code.
> The problem was it inherited the CloudStack DB schema and with it a 
> lot of cruft in dealing with being compatible with CloudStack.  So we 
> started a clean up of the code, which ended up being a huge refactor.  
> Our goal was to be able to support all of Go Daddy running on it.  
> That meant we needed a very flexible and extensible platform to 
> support a very heterogeneous environment.  (We had a very scalable L3 
> network design that I've yet to see any IaaS system be able to support 
> too.)  Additionally it needed to plug into all other systems in Go 
> Daddy for monitor, tracking, and network management (IPAM, etc).  This was by far my
favorite cloud to build.
> Really, really complex.  The goal was to build a system that could 
> deploy over 1 million VMs in less than a day.  (I think we did it too, 
> unfortunately I quit before I could fully prove it.  We got the system 
> to the point of deploying 1000 real VMs in 1 minute.  I just never had the time to fully
run the 1m test).
> So I've done a lot with cloud provisioning systems.  I was given an 
> incredible opportunity at Go Daddy that most people don't get.  I was 
> allowed to build an IaaS system from scratch and given millions and 
> millions in hardware to build and test it out and run real production 
> work loads on it.  I very much liked my cloud I built.  Unfortunately, 
> despite my best efforts, it was all proprietary.  I've decided to 
> abandon all the proprietary stuff I've done and try to do my best to 
> help out the open source IaaS landscape.  After looking at all the 
> stacks out there, I honestly believe CloudStack is the best one.  I reached out to Citrix
and they have been so gracious as to employee me full time to work on CloudStack.
> So expect to hear a lot from me.  I'm initially very interested at 
> improving the core architecture of CloudStack.  Just a couple random 
> things.  1) I'm here to code.  Talking is great, collaboration and 
> information sharing is important, but I'm not here just to talk.  I'm 
> here to actually get stuff done.  2)  I've only ran production clouds.  
> None of this dev/test/poc "optimize for innovation" crap.  I want a 
> system I can run in production.  So stability, upgrades, making 
> sysadmins happy, is very important to me.  3)  I'm very pragmatic and 
> practical about how I implement stuff.  I always have the 2 year 
> vision, but I'm very interested in what I can deliver in the next 3-6 
> months.  I enjoy delivering solutions far more than coding. 4) I'm really bad at typing
emails and make tons of typos and forget important words like "not" or "no" all the time.
> I really look forward to gaining the trust and respect of this 
> community and, more importantly, improving CloudStack.  I'm based out 
> of Phoenix, AZ area, if anybody happens to be in this fabulous part of 
> the country hit me up and we'll get together.
> Darren Shepherd
> [1] You can see my crappy slides for my OSCON talk at
> .  They 
> were really bad, basically I created them the night before.  Did I 
> mention I'm good at procrastination too.  Also that has the only known picture of me
on the internet.

View raw message