bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <n...@reactor8.com>
Subject RE: adding new deployment metadata files to bigtop
Date Wed, 13 Apr 2016 06:59:19 GMT
Some answers inline:

- the metadata files are presumably getting executed by some sort of external
  engine doing actual orchestration work, right? These files are, please
  forgive me for the lack of the better word or analogy, carry the
  function of topology map, and the actual installation/configuration of the
  nodes is still done by the good ol' Puppet. Is this correct?

Correct, meta file is for external usage, like a vagrant file which sits in bigtop dir, but
is only used externally by vagrant to deploy a node.  Difference is this instead of defining
a single node/vm/docker-image, etc, it’s typically defining a distributed topology/cluster,
and the workflow system would handle node provisioning and installing the bigtop packages.
 In regards to puppet question, right now there are two metadata descriptions, one that describes
topologies called an assembly, the other describes "components" which acts as a interface
def in bigtops case to the puppet modules so the system can load and execute the modules to
handle the config and service start part of it.

- with these files in place, how one can benefit from them? Is there a service
  that allow to plug them in? Or an orchestration engine, that could be ran on
  my laptop (or else)? Basically, I am trying to figure out what other pieces
  will be there and how they all get together.

I think in initial implementation the benefit would mostly be for people deploying distributed
clusters in environments like aws as the workflow system can handle automating the node deployments,
networking, and software install/config.  Instead of having just a single monolithic docker
image, or equivalent, can have something like "spark+hdfs+zeppelin cluster" with appropriate
master/worker design.  Right now the workflow system is bundled as starter uber docker all-in-one
container, kind of like the bigtop build image.

For local laptop/dev usage, will take little thinking/work, but we can probably look at making
a "localhost docker provider" that would be analogous to aws provider that could spin up and
execute the workflow all on localhost using docker for node deployment vs aws.


-----Original Message-----
From: Konstantin Boudnik [mailto:cos@apache.org] 
Sent: Wednesday, April 6, 2016 7:57 AM
To: user@bigtop.apache.org; dev@bigtop.apache.org
Subject: Re: adding new deployment metadata files to bigtop

Hi Nate.

Thanks for the decision to share the work you've been doing and help this community to advance.
I want to clarify a couple of questions here:
- the metadata files are presumably getting executed by some sort of external
  engine doing actual orchestration work, right? These files are, please
  forgive me for the lack of the better word or analogy, carry the
  function of topology map, and the actual installation/configuration of the
  nodes is still done by the good ol' Puppet. Is this correct?
- with these files in place, how one can benefit from them? Is there a service
  that allow to plug them in? Or an orchestration engine, that could be ran on
  my laptop (or else)? Basically, I am trying to figure out what other pieces
  will be there and how they all get together.

In general, I am very fond of having an orchestration in Bigtop, done one way or another.

Thanks!
  Cos

On Fri, Apr 01, 2016 at 05:31PM, nate@reactor8.com wrote:
> Hello bigtoppers,
> 
> Our org has been leveraging bigtop with our consulting work the past 
> couple years, primarily focused on deploying/managing clustered 
> services using the bigtop puppet modules.  We typically will pull 
> periodic updates and merge in latest bigtop puppet changes with our 
> version.  Recently we have had couple customer and community inquires 
> interested in using bigtop to start to play with various builds and 
> component versions, having us integrate our deployment stuff.  We are 
> initially going to maintain fork of bigtop to get the ball rolling on 
> a couple projects, but are interested in adding some base set of our 
> meta files to bigtop so people don't have to come to us for our fork, instead just maintain
and use vanilla apache bigtop.
> 
> The files we would like to contribute and maintain could be considered 
> akin to a vagrant or docker compose file, but instead of describing 
> single node/image, it describes a distributed service/cluster that can 
> run any number of services, leveraging the puppet modules for configuration.
> 
> Here is partial reference description for a spark service:
> 
> https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712
> 
> We are going to start prototyping in the next week or two, looking at 
> a file or two in the root folder, then probably unique folder in the 
> root that can house various service meta descriptions so people can 
> just use off the shelf without having to know details or write their 
> own by default.  In general the files are just text/yaml files, and 
> will have no disruption to existing gradle/tooling.
> 
> In meantime until we have something tangible, wanted to start the 
> conversation for community to provide any questions/feedback/etc about 
> approach of including a base set of service meta files


Mime
View raw message