cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Zhang <>
Subject RE: Introduction
Date Wed, 21 Aug 2013 22:59:28 GMT
Welcome back!

> -----Original Message-----
> From: Darren Shepherd []
> Sent: Wednesday, August 21, 2013 12:17 PM
> To:
> Subject: Introduction
>  All,
> I want to introduce myself to the Apache CloudStack community.  I've have had
> a long love/hate relationship with CloudStack over the years, but recently I've
> decided to fully dedicate myself to working on this platform (again).
> Just a little about me, well okay, this might be a little long.  I've been a Linux
> user for a good amount of time.  I've ran Linux as my primary OS for over 15
> years.  Both personally and professionally, my desktops/laptops have always
> ran Linux.  As soon as I discovered Linux I also fell in love with virtualization.
> I've ran the gamut in terms of virtualization technologies.  Type 1, Type 2,
> containers, etc.  I have a particular fondness for Xen though.  I've been a Xen
> user since the 2.x days.
> Couple years after I got into Linux I discovered programming.  After so many
> failed "./configure && make" I figured I'd should probably learn C/C++ to figure
> out why the heck nothing ever "make'd" right.  I worked for awhile running a
> lab for developers that were writing stuff in Java.  Again I learned Java to figure
> out why the heck their stuff kept failing to build and run.  After I learned Java I
> took the professional route of leaving the sysadmin stuff and started
> programming distributed systems.
> As means to an end I developed a system that would take code builds, deploy
> them to Xen thin provisioned VMs.  So a developer could go to a portal, click
> some stuff and get a clean VM that optionally had a code build on it.  We would
> then running automated integration tests against it.  This was around 2006.
> About a year or 2 later I discovered EC2 and was like "holy crap, that's exactly
> what I want."  Then I found eucalpytus and realized there was a whole industry
> around this stuff.  IaaS is basically the trifecta for me.  I get to combine my love
> of Linux, virtualization,
> and programming distributed system.   Basically since then I've been
> absolutely obsessed with IaaS.
> Around 2010 I got a job at Go Daddy specifically to build their public Cloud
> product.  Before I even got there they had already decided to go with
> vmops/  I'm pretty sure they were already when I started.
> At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java guys, 2 sysengs)
> we built three generations of clouds.  The first generation of cloud was
> basically CloudStack 2.2++.  It was CloudStack 2.2 with a new UI, billing system,
> storage subsystem, and a tons of operational tools.  That was the initial public
> launch of our Cloud.
> The second generation was "CloudStack like."  Basically we had a lot of
> problems with CloudStack at the time.  First, it was just a complete pain to
> operationalize.  Trying to hand off CloudStack to our ops and support teams was
> practically impossible.  Basically our level 1 and 2 support would be very
> efficient at escalating all issues to level 3 (my team).
> Typically the sysadmins would see something is stuck, they'd look at a log (that
> made no sense to them at all), see a 100 line stack trace and then they'd just
> toss the issue to the developers.  Also, their was no visibility to the general
> health of CloudStack.  If you asked ops, "How's production doing right now?"
> Their answer would be, "Well I can ping the UI and the java processes are still
> running."  In developer terms that is kinda like saying, "Well it compiles so it
> must work fine."
> Secondly, we had a hard time extending the platform.  We always seemed to
> be a little ahead of the curve in how we wanted to implement the cloud.
> How we wanted to run storage and networking didn't match at all what
> CloudStack was at that time.  So we embarked on improving it.  It was just way
> too difficult.  Basically we felt like 70% of our dev time was just fighting the
> framework.  So we made the choice to abandon the platform.  We started
> rewriting the the entire platform.  We did this piece meal over time.  We'd
> redo a subsystem, put that in production, and then move onto the next
> subsystem.  This ended up being very tricky because we had mixture of
> CloudStack and our own stuff.  We also always tried to do no downtime
> deployments.  So we would almost never lock people out of the UI and most
> deployments were during business hours.  We basically finished most of this
> work around summer last year.  I gave a talk at OSCON 2012 [1] regarding
> what we built.  A couple months later, solely based on business direction, we
> decided to stop selling the public cloud (it really wasn't shutdown until about
> May this year.  I'm proud to say it ran for almost 8 months with live customers,
> some quite large, with no incident and no dedicated ops team).
> After we shutdown the public cloud, we turned our effort to building a private
> cloud to run Go Daddy on top of it.  This marked the third generation of the
> cloud.  At this point the IaaS system was 100% our code.
> The problem was it inherited the CloudStack DB schema and with it a lot of
> cruft in dealing with being compatible with CloudStack.  So we started a clean
> up of the code, which ended up being a huge refactor.  Our goal was to be able
> to support all of Go Daddy running on it.  That meant we needed a very flexible
> and extensible platform to support a very heterogeneous environment.  (We
> had a very scalable L3 network design that I've yet to see any IaaS system be
> able to support too.)  Additionally it needed to plug into all other systems in Go
> Daddy for monitor, tracking, and network management (IPAM, etc).  This was
> by far my favorite cloud to build.
> Really, really complex.  The goal was to build a system that could deploy over 1
> million VMs in less than a day.  (I think we did it too, unfortunately I quit before
> I could fully prove it.  We got the system to the point of deploying 1000 real
> VMs in 1 minute.  I just never had the time to fully run the 1m test).
> So I've done a lot with cloud provisioning systems.  I was given an incredible
> opportunity at Go Daddy that most people don't get.  I was allowed to build an
> IaaS system from scratch and given millions and millions in hardware to build
> and test it out and run real production work loads on it.  I very much liked my
> cloud I built.  Unfortunately, despite my best efforts, it was all proprietary.  I've
> decided to abandon all the proprietary stuff I've done and try to do my best to
> help out the open source IaaS landscape.  After looking at all the stacks out
> there, I honestly believe CloudStack is the best one.  I reached out to Citrix and
> they have been so gracious as to employee me full time to work on CloudStack.
> So expect to hear a lot from me.  I'm initially very interested at improving the
> core architecture of CloudStack.  Just a couple random things.  1) I'm here to
> code.  Talking is great, collaboration and information sharing is important, but
> I'm not here just to talk.  I'm here to actually get stuff done.  2)  I've only ran
> production clouds.  None of this dev/test/poc "optimize for innovation" crap.  I
> want a system I can run in production.  So stability, upgrades, making
> sysadmins happy, is very important to me.  3)  I'm very pragmatic and practical
> about how I implement stuff.  I always have the 2 year vision, but I'm very
> interested in what I can deliver in the next 3-6 months.  I enjoy delivering
> solutions far more than coding. 4) I'm really bad at typing emails and make
> tons of typos and forget important words like "not" or "no" all the time.
> I really look forward to gaining the trust and respect of this community and,
> more importantly, improving CloudStack.  I'm based out of Phoenix, AZ area, if
> anybody happens to be in this fabulous part of the country hit me up and we'll
> get together.
> Darren Shepherd
> [1] You can see my crappy slides for my OSCON talk at
> .  They were
> really bad, basically I created them the night before.  Did I mention I'm good at
> procrastination too.  Also that has the only known picture of me on the internet.

View raw message