Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Message-ID: <5357E7E8.3050905@nanthrax.net>
Date: Wed, 23 Apr 2014 18:18:48 +0200
From: =?UTF-8?B?SmVhbi1CYXB0aXN0ZSBPbm9mcsOp?= <jb@nanthrax.net>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: general@incubator.apache.org
Subject: Re: [PROPOSAL] Brooklyn
References: <20140423142815.GX40448@Chips-MacBook-Air.local>
In-Reply-To: <20140423142815.GX40448@Chips-MacBook-Air.local>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

Hi,

it's interesting, and I see potential interaction with Apache Kalumet=20
(and ACE also as Marcel stated).

I would be pleased to be mentor and help on the project.

Regards
JB

On 04/23/2014 04:28 PM, Chip Childers wrote:
> The Brooklyn project [1] community is interested in moving to the ASF. =
 They
> have worked within their community to create a proposal [2] for the
> Incubator, and have managed to get agreement from 2 IPMC members to
> mentor them if accepted (Matt Hogstrom and Alex Karasulu).
>
> This is the discussion thread for this proposal.  We would also like to
> solicit contributions from additional IPMC members willing to mentor th=
e
> potential podling.  Anyone interested?
>
> -chip
>
> [1] http://brooklyncentral.github.io/
> [2] https://wiki.apache.org/incubator/BrooklynProposal
>
> Proposal follows:
>
> =3D Apache Brooklyn =3D
>
> =3D=3D Abstract =3D=3D
>
> Brooklyn is a framework for modeling, monitoring, and managing
> applications through autonomic blueprints.
>
> =3D=3D Proposal =3D=3D
>
> Brooklyn is about deploying and managing applications: composing a full
> stack for an application; deploying to cloud and non-cloud targets;
> using monitoring tools to collect key health/performance metrics;
> responding to situations such as a failing node; and adding or removing
> capacity to match demand.
>
> A Brooklyn blueprint defines an application, using a declarative YAML
> syntax supporting JVM plugins. A basic blueprint might comprise a singl=
e
> process, such as a web-application server running a WAR file or a SQL
> database and its associated DDL scripts. More complex blueprints
> encompass combinations of processes across multiple machines and
> services, such as a load-balancing HTTP server or SDN controller
> fronting a cluster of J2EE application servers, in turn connected to a
> resilient cluster of SQL database servers. Even larger clustered
> application running in multiple regions can be described, with features
> such as message buses with resilient brokers, cacheing tiers of NoSQL
> key-value store servers, a high-availability database cluster and
> multiple application components connected across these layers.
>
> One main benefit of these blueprints is that they are composable:
> best-practice blueprints for one process or pattern (e.g. a Cassandra
> cluster) can be incorporated in other blueprints (e.g. an application
> with a Cassandra cluster as one component). Another major benefit is
> that the blueprints can be treated as source code as part of an
> applications codebase: tested, tracked, versioned, and hardened as an
> integral part of the devops process. In some ways, Brooklyn is to
> run-time what Maven is to build-time.
>
> Brooklyn knows about Chef, Salt, and similar tools, and APT and Yum and
> plain old shell scripts, for deploying application components.
> Blueprints are built from a mixture of both off-the-shelf packages such
> as Tomcat, MySQL, Cassandra, and many more from our library; and
> components that are bespoke to individual applications; together with
> policies that allow the application to be self-managing.
>
> Brooklyn is built for the cloud, and will take a blueprint and deploy i=
t
> to one of many supported clouds or even to multiple different clouds, o=
r
> to private infrastructure (bring-your-own-node), or to other platforms.
> It will dynamically configure and connect all the different components
> of an application, e.g. so load balancers know where the web servers ar=
e
> and the web applications know where the database is.
>
> Brooklyn collects key metrics to monitor the health of applications; fo=
r
> example, by sending a request and measuring latency, or installing
> monitoring tools and using those to read a server=E2=80=99s management =
interface
> to determine the request queue length. These metrics can be fed into
> policies, which automatically take actions such as restarting a failed
> node, or scaling out the web tier if user demand exceeds capacity. This
> allows an application to be self-managing: to recover itself from simpl=
e
> failures, to scale out when demand increases and meet capacity; then
> scale in as demand drops and stop paying for spare capacity.
>
> In short, Brooklyn blueprints allow the best practices for deploying an=
d
> managing complex software to be codified as part of the software
> development process.
>
> =3D=3D=3D Background =3D=3D=3D
>
> Brooklyn is a product built from the ground up for application agility.
> This includes portability across non-cloud, cloud, and PaaS targets;
> devops-style infrastructure-as-code applied to applications; and
> real-time autonomic management based on promise theory. Some
> introductions to these concepts, associated tools, and open
> specifications may be useful.
>
> Cloud computing at its core is about provisioning resources on-demand.
> The most widely known aspect is IaaS (infrastructure-as-a-service) such
> as Amazon EC2, Softlayer, Google Cloud Platform, Apache CloudStack, or
> OpenStack. By leveraging the Apache jclouds project (and contributing
> heavily to it), the Brooklyn project is able to work with a large numbe=
r
> of such providers. Higher up the stack, however, there is an
> increasingly diverse set of platform targets, from PaaS
> (platform-as-a-service) such as Cloud Foundry and Apache Stratos,
> through to myriad containers and runtime fabrics such as LXC/Docker,
> Apache Mesos, Apache Karaf, Apache Hadoop, and Apache Usergrid and othe=
r
> backend-as-a-service environments. Brooklyn is based on the premise tha=
t
> applications may need to run in any or all of these, and the model must
> be flexible and open enough to support this.
>
> The buzzword-compliant trends of agile and devops have reinforced many
> important lessons:
>
>   * The truth is in the code (not any ancillary documents)
>   * If it isn't tested then assume it isn't working
>   * Toolchain integration and API's are key to a project's success
>   * Even more critical is empowering all stakeholders to a project
>   * Brooklyn's focus on blueprinting and modeling as code and API's
>     serves these principles.
>
> Another major influence on the design of Brooklyn are the ideas of
> autonomic computing and promise theory. It is not necessary to have a
> thorough understanding of these to use Brooklyn, but contributors tend
> to become versed in these ideas quickly. Essentially, autonomics is
> based on the concept of components looking after themselves where
> possible (self-healing, self-optimizing, etc), and exposing a sensor
> (data outputs) and effector (operations) API where they may need to
> controlled by another element. Promise theory extends this approach by
> introducing the idea that communicating intent (through promises) is a
> more reliable basis for complex cooperating systems than
> obligation-based effectors. Tools such as CF Engine, Chef, Puppet,
> Ansible, and Salt apply promise theory to files and processes on
> machines; Brooklyn can leverage all of these tools, complementing it
> with an application-oriented model.
>
> Finally we note some emerging standards in this area. OASIS CAMP (Cloud
> Application Management for Platforms) and TOSCA (Topology and
> Orchestration Specification for Cloud Applications) both define YAML
> application models similar to Brooklyn's. CAMP focuses on the REST API
> for interacting with such a management layer, and TOSCA focuses on
> declarative support for more sophisticated orchestration. Currently
> Brooklyn uses a YAML which complies with CAMP's syntax and exposes many
> of the CAMP REST API endpoints. We would like to support the
> hot-off-the-press TOSCA YAML and expand the CAMP REST API coverage.
>
> =3D=3D=3D Rationale =3D=3D=3D
>
> Building and deploying applications in the cloud computing era has
> changed many things. Provision a bare server just-in-time, and use
> automated tools to install an application. Use APIs to add the server t=
o
> a load balancer. When load goes up, provision another server; when load
> drops, kill a server to save money.
>
> Many new tools have appeared that take advantage of this new era.
> However each of them only solve part of the problem and don=E2=80=99t c=
onsider
> the big picture. For example, configuration management tools such as
> Chef can, in a single command, provision a new cloud server then instal=
l
> and configure an application -- but they require extra programming to
> reconfigure an load balancer whenever the pool of web servers changes.
> Amazon Auto Scaling can provision new servers and update load balancers=
,
> but it is dependent on CloudWatch -- this means either using proxy
> metrics such as average response time, or writing more code to expose a=
n
> application=E2=80=99s real metrics. A dedicated monitoring tool may be =
able to
> easily monitor the key metrics with little effort, but its alerts will
> need to be integrated it into the server provisioning process.
>
> So all the tools are there to to create and manage a cloud-scale
> application that can adapt to demand to meet user expectations without
> wasting money on superfluous services - but you will need several such
> tools and it is up to you to integrate them into your deployment plan.
> Some of these tools -- such as the Amazon Web Services web of EC2,
> CloudWatch, AutoScaling and CloudFormation -- mean that you may suffer
> from lock-in. Related projects in OpenStack (Heat, Ceilometer, Murano,
> Solum, etc) provide similar functionality but again for a restricted
> target. The most common policies (such as minimising request latency)
> may be easy, but less common policies such as follow-the-sun and
> follow-the-moon [1] may be up to you to implement. Your scaling policie=
s
> may understand that "high demand =3D add another server", but may not
> understand requirements such as some clustered services requiring an od=
d
> number of instances to prevent voting deadlocks.
>
> In this context the advantage of Brooklyn becomes apparent: a single
> tool is able to manage provisioning and application deployment, monitor
> an application=E2=80=99s health and metrics, understand the dependencie=
s between
> components (such as knowing that adding a new web server means that the
> load balancer needs reconfiguration) and apply complex policies to
> manage the application. The tool provides a REST API and a GUI, and
> allows the autonomic blueprints to be treated as an integral part of th=
e
> application. With Brooklyn, these policies become modular components
> which can be reused and easily added to blueprints.
>
> [1] Follow-the-sun refers to provisioning servers in network locations
> close to users. User activity levels typically correlates to the time o=
f
> day, with most activity during daylight hours and less during the night=
.
> Looking at a map showing daylight hours and server activity, demand
> patterns will tend to =E2=80=9Cfollow the sun=E2=80=9D. Follow-the-moon=
 refers to the
> converse, that many compute facilities will be under-utilised during th=
e
> night, and cheaper pricing may be available on spot markets at these
> times. Therefore, the location of the cheapest compute resources will
> tend to =E2=80=9Cfollow the moon=E2=80=9D.
>
> =3D=3D=3D Initial Goals =3D=3D=3D
>
> Brooklyn is a project in active development with some production
> deployments, although it is currently at version 0.7.0-M1.
>
> Our initial goals are to ensure that Brooklyn adopts the Apache Way, to
> grow the community, and to continue development towards a 1.0 release
> which makes a much-increased community very very happy.
>
> =3D=3D Current Status =3D=3D
>
> =3D=3D=3D Meritocracy =3D=3D=3D
>
> We firmly believe in meritocracy. Since it was open sourced in April
> 2012, Brooklyn has followed a published governance model based on
> several Apache projects where Brooklyn members have been active.
> Contributions can be made by anyone and are accepted after tests pass
> and at least one committer has reviewed it (and checked the CLAs).
> Contributors with a strong track record are invited to become committer=
s
> and join the management committee. We welcome the opportunity for
> talented developers to become committers and join the PMC.
>
> =3D=3D=3D Community =3D=3D=3D
>
> Brooklyn currently has a very active but small and homogenous community=
.
> The core team is aligned to a single corporation, with just a few code
> contributions externally. A larger and more diverse group of individual=
s
> and companies are using Brooklyn, building blueprint projects with
> Brooklyn dependenies, and discussing issues on the mailing list and in
> pull requests. These numbers are growing slowly.
>
> It is important that we foster and expand this community and attract
> more diversity to our core team.
>
> There are two routes to growing the community which are particularly
> noteworthy, and both are rendered more attractive by moving Brooklyn to
> Apache. ISV's and other projects can engage with the community as they
> require blueprinting capabilities, including several Apache projects fo=
r
> which Brooklyn blueprints already exist (see below). Secondly, some
> enterprises and large organizations are already using Brooklyn and are
> keen to see its future safeguarded by the Apache Foundation.
>
> =3D=3D=3D Core Developers =3D=3D=3D
>
> The following core developers are proposed, being the current set of
> committers to Brooklyn:
>
> Alex Heneveld, the creator of Brooklyn, is active with several open
> source projects (Apache jclouds, Apache Whirr, OpenStack) and standards
> groups (OASIS CAMP & TOSCA) in the area of application management. He
> co-founded Cloudsoft Corporation in 2009, where he currently serves as
> CTO.
>
> Aled Sage brings over a decade=E2=80=99s experience developing distribu=
ted
> applications, mostly in the enterprise sector but with an extensive lis=
t
> of open source project contributions. Particular areas of interest
> include concurrency, fault tolerance and transactional semantics.
>
> Andrew Kennedy is skilled in programming Java applications and services
> using modern open-source technologies. He has experience analysing,
> designing and implementing applications, middleware, messaging systems
> and databases, and is also an accomplished penetration tester, ethical
> hacker, network security engineer and security analyst. He is an ASF
> committer with the Apache Qpid project.
>
> Richard Downer has spent the last few years specialising in Cloud
> Computing. He has worked extensively with the OpenStack and CloudStack
> APIs, as well as Amazon EC2. Aside from Brooklyn, he has made numerous
> contributions to the Apache jclouds Java cloud abstraction project
> (http://jclouds.org).
>
> New core developers (initial committers) for the Brooklyn project will
> be welcomed based on the current criteria of code contributions and
> broad involvement in the community through code reviews, mailing lists,
> and/or IRC.
>
> =3D=3D=3D Alignment =3D=3D=3D
>
> We wish Brooklyn to follow the best practices of open source, and Apach=
e
> is a model of these practices. From the outset, Brooklyn has based
> itself on the Apache model for governance including making commits and
> voting on releases or substantive changes.
>
> Brooklyn is a heavy user of many Apache projects, and Brooklyn
> participants have been active in discussions and code contributions wit=
h
> several of these communities. In particular, Apache jclouds is our most
> important dependency. Apache CloudStack is a reliable and often-used
> deployment target for Brooklyn. Apache Maven is used to build Brooklyn
> and in some cases to resolve dependencies at runtime. In addition,
> Brooklyn support has been built for deploying and managing many Apache
> projects (Tomcat, Cassandra, Whirr, ActiveMQ, Kafka, and many others
> below). These relationships are discussed further below.
>
> =3D=3D Known Risks =3D=3D
>
> =3D=3D=3D Orphaned projects =3D=3D=3D
>
> The team behind Brooklyn firmly believe in their project and are
> committed to its success. They will be pushing the project until, and
> beyond, its community makes the project self-sustaining.
>
> =3D=3D=3D Inexperience with Open Source =3D=3D=3D
>
> Brooklyn has been open source since April 2012. All development has
> happened in public on GitHub[1], backed up by mailing lists on Google
> Groups[2][3] and IRC[4]. The team of committers are committed to the
> principles of open source, and count amongst their number existing
> Apache committers.
>
> For some of the extended group of contributors more used to commercial
> development, the commitment to openness is a habit that still needs to
> be learnt. For example, there is a feeling that some things need to be
> discussed "internally" first. However this instinct is being un-learned
> as our experience with the project in open source form continues. Under
> the guidance of the PPMC, we do not expect this to be a risk.
>
> [1] https://github.com/brooklyncentral/brooklyn
>
> [2] http://groups.google.com/group/brooklyn-users
>
> [3] http://groups.google.com/group/brooklyn-dev
>
> [4] #brooklyncentral on Freenode
>
> =3D=3D=3D Homogenous Developers =3D=3D=3D
>
> The initial committers are all affiliated with Cloudsoft Corporation Lt=
d
> and the vast majority of the commits to the project to date have been b=
y
> Cloudsoft employees. Some of the active participants are from
> organizations other than Cloudsoft and have expressed their support for
> the move to Apache and interest in continuing to contribute.
>
> We do have a geographically distributed list of contributors. Many are
> in the United Kingdom, but there are a significant number of commits
> from people from other European countries and from the United States.
>
> As noted previously, our community is beginning to grow and we look
> forward to bringing more diversity into the list of committers and PPMC
> members. However we recognize the current lack of diversity as a known
> risk.
>
> =3D=3D=3D Reliance on Salaried Developers =3D=3D=3D
>
> The initial committers are all affiliated with Cloudsoft Corporation
> Ltd. The vast majority of the commits to the project to date have been
> by Cloudsoft employees in the line of their work.
>
> Our community is beginning to grow, and we hope to grow the community
> significantly and bring more diversity into the list of committers and
> PPMC members. However we recognise the reliance on salaried developers
> as a known risk.
>
> =3D=3D=3D Relationships with Other Apache Products =3D=3D=3D
>
> Brooklyn has relationships to several other Apache projects:
>
> Brooklyn has entities for many Apache projects; for example, it can
> deploy instances of Cassandra, Tomcat, HTTP Server, ActiveMQ, Qpid,
> Solr, Storm, Karaf, Kafka, Zookeeper and CouchDB
> Brooklyn uses Whirr to deploy instances of Hadoop and HBase
> jclouds is our single most important dependency, which we use for
> virtually all our interaction with cloud provider APIs.
> CloudStack is a first-class target for Brooklyn application deployments=
,
> and we have used it on many occasions.
> We use Maven, HttpComponents and various Commons projects.
> There is some overlap with the Whirr project; at face value, both Whirr
> and Brooklyn allow complex applications to be deployed into cloud
> providers. However our emphasis is different -- Brooklyn provides
> policy-based management to monitor applications. We consider Brooklyn t=
o
> be complementary rather than competitive - Brooklyn supports the use of
> Whirr to deploy Hadoop clusters.
>
> =3D=3D=3D A Excessive Fascination with the Apache Brand =3D=3D=3D
>
> We consider that Apache is a natural home for Brooklyn; we license our
> code under the Apache License, and we make extensive use of jclouds
> which recently graduated from the Apache incubator; we aspire to follow
> open source best practices. Our proposal to the Apache Incubator is
> based in pragmatism, not idolatry.
>
> =3D=3D Documentation =3D=3D
>
> Brooklyn's primary website is available at http://brooklyn.io/
>
> =3D=3D Initial Source =3D=3D
>
> There is an established existing code base for Brooklyn located at
> GitHub: https://github.com/brooklyncentral/brooklyn
>
> =3D=3D Source and Intellectual Property Submission Plan =3D=3D
>
> The code is licensed under Apache License V2, and all contributions are
> subject to an Apache-style ICLA with Cloudsoft Corporation Ltd. In
> addition to the code, Cloudsoft has registered a trademark on the name
> "Brooklyn", and the logo currently used for the project. The PPMC will
> work with Cloudsoft Corporation and its stakeholders in order to
> identify these properties and donate them to the Apache Foundation.
>
> There are also a number of other assets related to the project, such as
> its domain name, Twitter account, and IRC channel. During incubation th=
e
> PPMC will identify all these assets, and arrange the transfer,
> replacement, or deprecation of these assets as appropriate.
>
> =3D=3D External Dependencies =3D=3D
>
> The vast majority of Brooklyn's direct and indirect dependencies are
> licensed (or dual-licensed) under The Apache License, Category A or B
> licenses, or are dedicated to the public domain.
>
> A small number dependencies give cause for concern; there are two cases
> of LGPL dependencies, and a number of dependencies where the license is
> either unknown, or compatibility with Apache is unknown.
>
> The PPMC must therefore:
>
>   * determine the license for all code where it is not currently known;
>   * determine ASL compatibility where this is not currently known;
>   * replace all dependencies that do not have an ASL compatible license=
.
>
> =3D=3D Cryptography =3D=3D
>
> Brooklyn does not directly implement cryptography code. It may rely on
> other projects to perform certain cryptographic tasks (e.g. calculating
> signatures for requests to cloud providers.)
>
> =3D=3D Required Resources =3D=3D
>
> =3D=3D=3D Mailing lists =3D=3D=3D
>
> We seek "dev" and "user" lists to replace the lists currently in use. I=
n
> alignment with Apache's standard practices, we would also have a
> "private" list for the PMC members, and a "commits" list.
>
> =3D=3D=3D Source Control =3D=3D=3D
>
> We would require a Git repository named "brooklyn" to hold the Brooklyn
> source code, and "brooklyn-site" to hold the web site source code. We
> would like these to be mirrored to GitHub repositories, where we intend
> to follow the same model currently used by Apache jclouds.
>
> =3D=3D=3D Issue Tracking =3D=3D=3D
>
> Jira, with a project name of "BROOKLYN".
>
> =3D=3D=3D Other Resources =3D=3D=3D
>
> No other resources are required at this time.
>
> =3D=3D Initial Committers =3D=3D
>
>   * Alex Heneveld
>   * Aled Sage
>   * Andrew Kennedy grkvlt@apache.org
>   * Richard Downer
>
> =3D=3D Affiliations =3D=3D
>
> The majority of the commits to the project so far have been made by
> people who were employees of Cloudsoft Corporation Ltd at the time of
> the contribution, and the vast majority of those commits were in the
> line of their employment. All named initial committers are employees of
> Cloudsoft.
>
> As stated in the Known Risks section, we appreciate that this cannot
> continue long term, and a key goal of the podling will be to encourage
> people not affiliated with Cloudsoft to join the project as committers
> and PMC members.
>
> =3D=3D Sponsors =3D=3D
>
> Champion
>
>   * Chip Childers chipchilders@apache.org
>
> Nominated Mentors
>
>   * Matt Hogstrom hogstrom@apache.org
>   * Alex Karasulu akarasulu@apache.org
>
> We appreciate that 3-5 Mentors greatly assists the incubation process,
> so welcome more Mentors.
>
> Sponsoring Entity
>
> We are asking that the Incubator sponsors the Brooklyn podling.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

--=20
Jean-Baptiste Onofr=C3=A9
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org