bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jay vyas <>
Subject Re: newbie question of BigTop
Date Tue, 09 Jun 2015 02:01:56 GMT
also, not sure if i put this link up for you yet, but here : .
its a bigtop internals explanation which shows some info about how the
vagrant and puppet stuff works.
you will be ready to hack bigtop w/ the best of em after watching that :)

On Mon, Jun 8, 2015 at 8:02 PM, Demai Ni <> wrote:

> Evans,
> thank you very much for the detail instructions.
> I understood that the cluster need to be passwordless ssh or someway to
> provide userid/password of all nodes. So that actions can be propogated to
> other nodes.
> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
> would do such job for end-users. But the document didn't mention such
> tool(or I missed them?), and I got the impression that that steps 1)~3) are
> required by manual effort.
> Maybe those tools are common-sense for an experienced cluster administer,
> so no document is necessary.:-) Anyway, now that with your help, I will try
> them out. Thanks
> Demai
> On Sat, Jun 6, 2015 at 12:03 PM, Evans Ye <> wrote:
>> The missing note in the document is that the cluster admin need to setup
>> some general environment settings such as
>> the password-less ssh. Definitely it would be great if we can provide some
>> suggestions or ways to do that, but I think somehow you still need to setup
>> that during machine provisioning(installing OS, setting up hostname,
>> ip, password, ssh keys), right?
>> If you have bunch of machines ready for the deployment, here's the step
>> to setup a bigtop cluster:
>> 1) install puppet on each node
>> 2) install puppet modules on each node
>> puppet apply --modulepath= -e "include bigtop_toolchain::puppet-modules"
>> 2) propagate puppet recipes and configurations
>> <>on
>> each node
>> 3) run puppet apply on each node
>> 4) done!
>> The puppet and module installation can be done through cluster admin tool
>> such as pdsh, which allows you to run same command across the whole
>> cluster in parallel mode. The puppet recipes or bigtop repo can be put
>> in a share storage such as NFS so that it is available for all nodes. After
>> all of this are set, you can then run puppet apply on each node through
>> pdsh as well. This should give you an up and running bigtop cluster.
>> If you're not quite sure how to do that, you can also refer to bigtop's vagrant
>> provisioner
>> <>,
>> which demonstrate the whole process to setup a bigtop cluster by a click of
>> button.
>> Of course, you can ask questions in mailing list when you encounter any
>> issue. We're happy to help. :)
>> 2015-06-06 6:30 GMT+08:00 Demai Ni <>:
>>> hi, folks,
>>> I am looking at BigTop as noncommercial hadoop distribution package, as
>>> an alternative of CDH, HDP. And this is my first time to try out bigTop.
>>> My expectation was:
>>> 1) setup my cluster with passwordless ssh
>>> 2) download the necessary bigtop components on one of the nodes.
>>> 3) run an installation/configure tool from the node on 2)
>>> 4) BigTop will (through puppet) install and configure the whole cluster.
>>> the point is that no need of a lot of manual work on every node.
>>> But I am following instruction for 0.8.0 and 0.7.0, like
>>> there are quite a few steps, (almost all the steps in above link) are
>>> required to be performed on each node?
>>> so it looks like the human work is significant higher between small(3
>>> node) and large(100+) cluster.  Is that right?
>>> Thanks.
>>> Demai

jay vyas

View raw message