bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Starina <david.star...@gmail.com>
Subject Re: User guide?
Date Wed, 16 Mar 2016 22:39:48 GMT
Hi,

Roman, Evans, thank you for the tips - that helped a lot. Still have some
problems though, an error when running puppet apply ...

This is a sample of errors I got:








*Debug: Executing '/bin/yum -d 0 -e 0 -y install
ignite-hadoop-service'Error: Could not update: Execution of '/bin/yum -d 0
-e 0 -y install ignite-hadoop-service' returned 1: Error: Package:
ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
osgi(org.apache.servicemix.bundles.spring-beans)Error: Package:
ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
osgi(org.apache.servicemix.bundles.spring-context-support)Error: Package:
ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package:
ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
osgi(org.apache.servicemix.bundles.spring-core)*
.....

Also, after that:






*Notice:
/Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
Dependency Package[ignite-hadoop] has failures: trueWarning:
/Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
Skipping because of failed dependenciesNotice:
/Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency
Package[openjdk-7-jre-headless] has failures: trueWarning:
/Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Skipping
because of failed dependenciesNotice:
/Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
Dependency Package[openjdk-7-jre-headless] has failures: trueWarning:
/Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]: Skipping
because of failed dependenciesNotice:
/Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/ignite-hadoop/conf/default-config.xml]:
Dependency Package[ignite-hadoop] has failures: true*
.....


This is the "site.yaml" I used ("hadoop" wasn't listed in components, but I
think it should be there?)















*bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs:  -
/data/1hadoop_cluster_node::cluster_components:  - hadoop  - ignite_hadoop
- hue  - mahout  - yarn  - zookeeperbigtop::jdk_package_name:
"openjdk-7-jre-headless"bigtop::bigtop_repo_uri:
"http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
<http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64>"*




Any idea what I am doing wrong?

--David

On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <evansye@apache.org> wrote:

> David
>
> Most of the time you need to run same command on all nodes. "pdsh" would
> be a good tool to use.
>
> 1. You need to have puppet agent installed on every node. yum -y install
> puppet will do the trick, or you need to download repo, or use
> bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's no deploy effort
> other than just install the package.
>
> 2. Masterless means each puppet agent takes commands and runs
> independently.  We DONT use puppet master, which is a centralized master
> daemon to dispatch works to all the registered puppet agent.
>
> 3. Next, you need to copy bigtop puppet recipes and bigtop hiera
> configuration to all nodes, or at least make them available for all nodes
> via nfs.
>
> 4. Finally, execute the puppet apply command on all nodes and the puppet
> recipes will automatically figure out it's role. There no need to deploy
> head node first and then slaves.
>
> Ask if you still have questions.
>
>
> Thanks
>
> Evans
> 2016年3月15日 下午4:10於 "David Starina" <david.starina@gmail.com>寫道:
>
>> Hi guys,
>>
>> me again. I haven't set up Puppet before, can you just point me in the
>> right direction? I know this is not a Puppet mailing list, but I bet you
>> have some tips for me :-)
>>
>> So what do I need to do? Install Puppet on every node? What does it mean
>> that the mode of Puppet is "*masterless"* - do I just have to do a basic
>> installation of Puppet and not care much about setting it up, and then
>> deploy bigtop on every node separately? And when setting the Hadoop head
>> node - does that mean that I need to deploy Bigtop on this node first, and
>> Bigtop will know this is the master node by the url or IP I provide to it?
>> And then deploy Bigtop via Puppet on all the other nodes and they will
>> connect to the head node (which is the NameNode, I suppose?).
>>
>> I can do this via trial and error, but if you guys can help me reduce the
>> "error" part, that will be great :) Sorry for such beginner questions, I
>> haven't used Puppet before ... The reason I'm asking is, I got access to a
>> "physical" (actually virtual, but it's the same) environment for a limited
>> amount of time (two weeks), and I need to start testing on it asap - and
>> since my day job takes a lot of my time, if I wanted to do this without
>> help it could take me two weeks just to set up the environment - but I need
>> to install it, which is not the central part of my project, and then run
>> some ML algorithms on a text corpus, which will be the main part of the
>> project ...
>>
>> --David
>>
>> On Thu, Mar 10, 2016 at 11:28 AM, David Starina <david.starina@gmail.com>
>> wrote:
>>
>>> Thanks for the info!
>>>
>>> --David
>>>
>>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas <jayunit100.apache@gmail.com>
>>> wrote:
>>>
>>>> I think there is enough info on youtube that you can cobble together a
>>>> syllabus if you need structured learning.
>>>>
>>>> See cos/Romans Bigtop talks which are on YouTube and then my Bigtop
>>>> internals hackathon video.   That gives you both the high and low level
>>>> details to do whatever you want.
>>>>
>>>> > On Mar 8, 2016, at 1:11 PM, Konstantin Boudnik <cos@apache.org>
>>>> wrote:
>>>> >
>>>> > We also had this discussion to start producing "The Big Book of Apache
>>>> > Bigtop" like HBase does, but never got into the materialization part
>>>> of that
>>>> > project.
>>>> >
>>>> > Cos
>>>> >
>>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans Ye wrote:
>>>> >> Hi David,
>>>> >>
>>>> >> If you just want to take Bigtop binaries to deploy a cluster, you
>>>> can refer
>>>> >> to our puppet management recipes:
>>>> >> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet
>>>> >>
>>>> >> BTW bigtop provisioner is the best source to reference for how bigtop
>>>> >> provision a cluster:
>>>> >>
>>>> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide
>>>> >>
>>>> >> Please ask us if you find anything confusing.
>>>> >>
>>>> >>
>>>> >> Evans
>>>> >>
>>>> >>
>>>> >>
>>>> >> 2016-03-08 0:48 GMT+08:00 David Starina <david.starina@gmail.com>:
>>>> >>
>>>> >>> Hi,
>>>> >>>
>>>> >>> Is there some version of Bigtop User Guide available? I know
it
>>>> used to be
>>>> >>> on the be on http://bigtop.apache.org/ under the "User Guide"
>>>> link, but
>>>> >>> there's nothing there anymore ...
>>>> >>>
>>>> >>> I need some guide on how to install Bigtop on a physical cluster
>>>> ... It
>>>> >>> was easy to create a virtual cluster on my machine using Vagrant,
>>>> but I
>>>> >>> cannot find a manual to install on a physical cluster ...
>>>> >>>
>>>> >>> Cheers,
>>>> >>> David
>>>> >>>
>>>>
>>>
>>>
>>

Mime
View raw message