Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of kelven.yang@citrix.com
 designates 66.165.176.89 as permitted sender)
From: Kelven Yang <kelven.yang@citrix.com>
To: "cloudstack-dev@incubator.apache.org"
	<cloudstack-dev@incubator.apache.org>
Date: Wed, 8 Aug 2012 22:38:48 -0700
Subject: Re: [RFC] CloudStack Architectural Refactor
Thread-Topic: [RFC] CloudStack Architectural Refactor
Thread-Index: Ac118UH0sGtZ+nzEQMKO59TMtgam7Q==
Message-ID: <CC4898F0.78FB%kelven.yang@citrix.com>
In-Reply-To: 
 <B1DF26ECC0458748AC97CECE2DA98D41012F9EB380A5@SJCPMAILBOX01.citrite.net>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.2.3.120616
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

As part of the effort, a very low level service about job control will be
refactored to support better CloudStack operation.

Like process to an operating system, jobs are "process" in a cloud
platform, we are going to improve the job management with following things
in mind


=20
		1) Be able to schedule timely-run jobs
		2) Job heartbeat/progress report and time-out management
		3) Job cancellation
		4) job scheduling affinity
			Sub job can be scheduled to run within the same container of its parent
job (easier for diagnostic)

		5) Sub job and job synchronization
			Job can be suspended and resumed based on event notification, the event
should be a object state change (for example, sub job enters into
completed state). this will make high level work flow control possible.
		=09

		6) Tracking diagnostic
Root job should either be system initiated or user initiated, we will
incorporate with logging facility to log all root job info, this will help
solve diagnostic and troubleshooting headaches

		7) Job context and job persistency
			Restart job execution across sessions.

		8) Job throttling support
			Provide mechanism to allow job executors to perform proactive load
management

Kelven


On 8/8/12 9:38 PM, "Alex Huang" <Alex.Huang@citrix.com> wrote:

>CloudStack 3.x has quite a bit of baggage after years of development.
>This makes it difficult for new committers to join the project.  Even for
>existing committers, it is easy to make mistakes.  We're looking to make
>some fundamental changes to CloudStack to overcome these problems.
>Please feel free to comment.
>
>Problems:
>  - No clear separation between orchestration and the subsystems being
>orchestrated.  We've done a lot of work in this area in networking.
>However, storage and hypervisor  are still missing.
>  - No clear separation between orchestration and the ACL system.
>  - No clear separation between orchestration and the extra services
>(read SSVM, console proxy VM)
>  - CloudStack believes it is the master of all physical resources when
>it is not.
>
>Goals:
>  - Form a core orchestration engine that is free of unnecessary
>artifacts such as ACL.
>  - Clear definition of functionalities and how to contribute to
>different parts of CloudStack.
>  - Allow developers to write and deploy services on the CloudStack
>orchestration server.
>
>Design:
>CloudStack will be refactor into the following functional areas.  Details
>to follow after the design.
>Orchestration Platform will include the following:
>  - Orchestration Framework:  Orchestration engine is key in cloud
>infrastructure.  =20
>  - Subsystem Plugins: Actual implementations in network, hypervisor, and
>storage of the orchestration steps.
>  - Services Framework: A framework for others developers to build
>additional services that can be launched using the Orchestration Service.
>  - Events Framework: Allow third-party components to integrate with
>Orchestration Platform through events only.
>
>On top of the Orchestration Platform, CloudStack will come with the
>following Services which can be enabled/disabled without affecting
>CloudStack orchestration functionalities.  Each of these services can be
>deployed on bare metal, VM, or as a webapp.
>  - ACL Service to provide advance ACL control to other services launched
>  - End User API Service to provide orchestration of end user VMs
>  - ConsoleProxy Service to provide console access
>  - Template Service to provide template/ISO transport and access within
>a zone
>  - Router Service to provide routing services to end user VMs
>  - Object Store Service to provide object store functionality
>  - EC2 API Service
>
>Details:
>Orchestration Platform
>  - Continues to manage hardware resources
>  - Provide administrative API/UI
>  - Minimal ACL provided
>  - Provides a Services Directory for services to find each other
>  - Includes the following components
>
>Orchestration Framework
>  - Orchestrates VMs, Volumes, Templates, Networks operations within a
>data center
>  - Ensures the operations are idempotent
>  - Deals with hiccups and maintenances needs of the system administrator
>so that services built on top do not have to.
>  - Breaks down cloud operations into steps and pushes the steps to the
>subsystems to be perform but does not actually perform the steps.
>
>Subsystem Plugins:
>  - Plugins that knows how to perform different functionalities that the
>orchestration framework needs.
>  - Current Subsystem Planned: Deployment Planning, NetworkGuru,
>NetworkElement, Storage, Snapshot, Backup (see Edison's RFC)
>
>Services Framework:  (RFC upcoming)
>  - Leverages the current system VM to allow more services to be built.
>
>Events Framework: (RFC upcoming)
>  - Publishes all Orchestration platform entity changes as events.
>  - Allow other components outside of Orchestration Platform to interact
>with Orchestration Platform only through event notifications.
>
>Design Thoughts:
>  - In this design, end user API and end user VMs is one particular
>service provided.  End User VM services can innovate independently from
>the Orchestration Platform.
>  - Domains and Accounts will reside with the ACL service which can
>integrate with Active Directory and LDAP services.
>
>--Alex