bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jay vyas <jayunit100.apa...@gmail.com>
Subject Re: newbie question of BigTop
Date Tue, 09 Jun 2015 02:01:56 GMT
also, not sure if i put this link up for you yet, but here :
https://www.youtube.com/watch?v=4GfcKEjO6e8&feature=youtu.be .
its a bigtop internals explanation which shows some info about how the
vagrant and puppet stuff works.
you will be ready to hack bigtop w/ the best of em after watching that :)

On Mon, Jun 8, 2015 at 8:02 PM, Demai Ni <nidmgg@gmail.com> wrote:

> Evans,
>
> thank you very much for the detail instructions.
>
> I understood that the cluster need to be passwordless ssh or someway to
> provide userid/password of all nodes. So that actions can be propogated to
> other nodes.
>
> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
> would do such job for end-users. But the document didn't mention such
> tool(or I missed them?), and I got the impression that that steps 1)~3) are
> required by manual effort.
>
> Maybe those tools are common-sense for an experienced cluster administer,
> so no document is necessary.:-) Anyway, now that with your help, I will try
> them out. Thanks
>
> Demai
>
> On Sat, Jun 6, 2015 at 12:03 PM, Evans Ye <evansye@apache.org> wrote:
>
>> The missing note in the document is that the cluster admin need to setup
>> some general environment settings such as
>> the password-less ssh. Definitely it would be great if we can provide some
>> suggestions or ways to do that, but I think somehow you still need to setup
>> that during machine provisioning(installing OS, setting up hostname,
>> ip, password, ssh keys), right?
>>
>> If you have bunch of machines ready for the deployment, here's the step
>> to setup a bigtop cluster:
>>
>> 1) install puppet on each node
>> 2) install puppet modules on each node
>>
>> puppet apply --modulepath= -e "include bigtop_toolchain::puppet-modules"
>>
>> 2) propagate puppet recipes and configurations
>> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet>on
>> each node
>> 3) run puppet apply on each node
>> 4) done!
>>
>> The puppet and module installation can be done through cluster admin tool
>> such as pdsh, which allows you to run same command across the whole
>> cluster in parallel mode. The puppet recipes or bigtop repo can be put
>> in a share storage such as NFS so that it is available for all nodes. After
>> all of this are set, you can then run puppet apply on each node through
>> pdsh as well. This should give you an up and running bigtop cluster.
>>
>> If you're not quite sure how to do that, you can also refer to bigtop's vagrant
>> provisioner
>> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet-vm>,
>> which demonstrate the whole process to setup a bigtop cluster by a click of
>> button.
>>
>> Of course, you can ask questions in mailing list when you encounter any
>> issue. We're happy to help. :)
>>
>> 2015-06-06 6:30 GMT+08:00 Demai Ni <nidmgg@gmail.com>:
>>
>>> hi, folks,
>>>
>>> I am looking at BigTop as noncommercial hadoop distribution package, as
>>> an alternative of CDH, HDP. And this is my first time to try out bigTop.
>>>
>>> My expectation was:
>>> 1) setup my cluster with passwordless ssh
>>> 2) download the necessary bigtop components on one of the nodes.
>>> 3) run an installation/configure tool from the node on 2)
>>> 4) BigTop will (through puppet) install and configure the whole cluster.
>>>
>>> the point is that no need of a lot of manual work on every node.
>>>
>>> But I am following instruction for 0.8.0 and 0.7.0, like
>>>
>>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
>>>
>>> there are quite a few steps, (almost all the steps in above link) are
>>> required to be performed on each node?
>>>
>>> so it looks like the human work is significant higher between small(3
>>> node) and large(100+) cluster.  Is that right?
>>>
>>> Thanks.
>>>
>>> Demai
>>>
>>>
>>
>>
>


-- 
jay vyas

Mime
View raw message