bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Starina <david.star...@gmail.com>
Subject Re: User guide?
Date Thu, 17 Mar 2016 11:46:02 GMT
Konstantin,

So I should be safer using CentOS 6?

Also - was I right to include "hadoop" in site.yaml components? It's not
listed in the basic site.yaml, but I am guessing it is not installed
automatically?

-David

On Thu, Mar 17, 2016 at 7:17 AM, David Starina <david.starina@gmail.com>
wrote:

> That's right, I'm using CentOS 7.
>
> --David
>
>
> On Thursday, 17 March 2016, Konstantin Boudnik <cos@apache.org> wrote:
>
>> Hmm... that's plain nuts... Looks like latest ignite is pulling in some
>> crap
>> like osgi... OMG, WTF?
>>
>> Thanks for discovering this David! Somehow it never got stepped upon in my
>> tests. Are you on centos7?
>>
>> Cos
>>
>> On Wed, Mar 16, 2016 at 11:39PM, David Starina wrote:
>> > Hi,
>> >
>> > Roman, Evans, thank you for the tips - that helped a lot. Still have
>> some
>> > problems though, an error when running puppet apply ...
>> >
>> > This is a sample of errors I got:
>> >
>> >
>> > *Debug: Executing '/bin/yum -d 0 -e 0 -y install
>> > ignite-hadoop-service'Error: Could not update: Execution of '/bin/yum
>> -d 0
>> > -e 0 -y install ignite-hadoop-service' returned 1: Error: Package:
>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>  Requires:
>> > osgi(org.apache.servicemix.bundles.spring-beans)Error: Package:
>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>  Requires:
>> > osgi(org.apache.servicemix.bundles.spring-context-support)Error:
>> Package:
>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>  Requires:
>> > osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package:
>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>  Requires:
>> > osgi(org.apache.servicemix.bundles.spring-core)*
>> > .....
>> >
>> > Also, after that:
>> >
>> >
>> >
>> >
>> >
>> >
>> > *Notice:
>> >
>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
>> > Dependency Package[ignite-hadoop] has failures: trueWarning:
>> >
>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
>> > Skipping because of failed dependenciesNotice:
>> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency
>> > Package[openjdk-7-jre-headless] has failures: trueWarning:
>> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Skipping
>> > because of failed dependenciesNotice:
>> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
>> > Dependency Package[openjdk-7-jre-headless] has failures: trueWarning:
>> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
>> Skipping
>> > because of failed dependenciesNotice:
>> >
>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/ignite-hadoop/conf/default-config.xml]:
>> > Dependency Package[ignite-hadoop] has failures: true*
>> > .....
>> >
>> >
>> > This is the "site.yaml" I used ("hadoop" wasn't listed in components,
>> but I
>> > think it should be there?)
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs:  -
>> > /data/1hadoop_cluster_node::cluster_components:  - hadoop  -
>> ignite_hadoop
>> > - hue  - mahout  - yarn  - zookeeperbigtop::jdk_package_name:
>> > "openjdk-7-jre-headless"bigtop::bigtop_repo_uri:
>> > "http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
>> > <http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64>"*
>> >
>> >
>> >
>> >
>> > Any idea what I am doing wrong?
>> >
>> > --David
>> >
>> > On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <evansye@apache.org> wrote:
>> >
>> > > David
>> > >
>> > > Most of the time you need to run same command on all nodes. "pdsh"
>> would
>> > > be a good tool to use.
>> > >
>> > > 1. You need to have puppet agent installed on every node. yum -y
>> install
>> > > puppet will do the trick, or you need to download repo, or use
>> > > bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's no deploy
>> effort
>> > > other than just install the package.
>> > >
>> > > 2. Masterless means each puppet agent takes commands and runs
>> > > independently.  We DONT use puppet master, which is a centralized
>> master
>> > > daemon to dispatch works to all the registered puppet agent.
>> > >
>> > > 3. Next, you need to copy bigtop puppet recipes and bigtop hiera
>> > > configuration to all nodes, or at least make them available for all
>> nodes
>> > > via nfs.
>> > >
>> > > 4. Finally, execute the puppet apply command on all nodes and the
>> puppet
>> > > recipes will automatically figure out it's role. There no need to
>> deploy
>> > > head node first and then slaves.
>> > >
>> > > Ask if you still have questions.
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Evans
>> > > 2016年3月15日 下午4:10於 "David Starina" <david.starina@gmail.com>寫道:
>> > >
>> > >> Hi guys,
>> > >>
>> > >> me again. I haven't set up Puppet before, can you just point me in
>> the
>> > >> right direction? I know this is not a Puppet mailing list, but I bet
>> you
>> > >> have some tips for me :-)
>> > >>
>> > >> So what do I need to do? Install Puppet on every node? What does it
>> mean
>> > >> that the mode of Puppet is "*masterless"* - do I just have to do a
>> basic
>> > >> installation of Puppet and not care much about setting it up, and
>> then
>> > >> deploy bigtop on every node separately? And when setting the Hadoop
>> head
>> > >> node - does that mean that I need to deploy Bigtop on this node
>> first, and
>> > >> Bigtop will know this is the master node by the url or IP I provide
>> to it?
>> > >> And then deploy Bigtop via Puppet on all the other nodes and they
>> will
>> > >> connect to the head node (which is the NameNode, I suppose?).
>> > >>
>> > >> I can do this via trial and error, but if you guys can help me
>> reduce the
>> > >> "error" part, that will be great :) Sorry for such beginner
>> questions, I
>> > >> haven't used Puppet before ... The reason I'm asking is, I got
>> access to a
>> > >> "physical" (actually virtual, but it's the same) environment for a
>> limited
>> > >> amount of time (two weeks), and I need to start testing on it asap
-
>> and
>> > >> since my day job takes a lot of my time, if I wanted to do this
>> without
>> > >> help it could take me two weeks just to set up the environment - but
>> I need
>> > >> to install it, which is not the central part of my project, and then
>> run
>> > >> some ML algorithms on a text corpus, which will be the main part of
>> the
>> > >> project ...
>> > >>
>> > >> --David
>> > >>
>> > >> On Thu, Mar 10, 2016 at 11:28 AM, David Starina <
>> david.starina@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> Thanks for the info!
>> > >>>
>> > >>> --David
>> > >>>
>> > >>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas <
>> jayunit100.apache@gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> I think there is enough info on youtube that you can cobble
>> together a
>> > >>>> syllabus if you need structured learning.
>> > >>>>
>> > >>>> See cos/Romans Bigtop talks which are on YouTube and then my
Bigtop
>> > >>>> internals hackathon video.   That gives you both the high and
low
>> level
>> > >>>> details to do whatever you want.
>> > >>>>
>> > >>>> > On Mar 8, 2016, at 1:11 PM, Konstantin Boudnik <cos@apache.org>
>> > >>>> wrote:
>> > >>>> >
>> > >>>> > We also had this discussion to start producing "The Big
Book of
>> Apache
>> > >>>> > Bigtop" like HBase does, but never got into the materialization
>> part
>> > >>>> of that
>> > >>>> > project.
>> > >>>> >
>> > >>>> > Cos
>> > >>>> >
>> > >>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans Ye wrote:
>> > >>>> >> Hi David,
>> > >>>> >>
>> > >>>> >> If you just want to take Bigtop binaries to deploy
a cluster,
>> you
>> > >>>> can refer
>> > >>>> >> to our puppet management recipes:
>> > >>>> >>
>> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet
>> > >>>> >>
>> > >>>> >> BTW bigtop provisioner is the best source to reference
for how
>> bigtop
>> > >>>> >> provision a cluster:
>> > >>>> >>
>> > >>>>
>> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide
>> > >>>> >>
>> > >>>> >> Please ask us if you find anything confusing.
>> > >>>> >>
>> > >>>> >>
>> > >>>> >> Evans
>> > >>>> >>
>> > >>>> >>
>> > >>>> >>
>> > >>>> >> 2016-03-08 0:48 GMT+08:00 David Starina <
>> david.starina@gmail.com>:
>> > >>>> >>
>> > >>>> >>> Hi,
>> > >>>> >>>
>> > >>>> >>> Is there some version of Bigtop User Guide available?
I know it
>> > >>>> used to be
>> > >>>> >>> on the be on http://bigtop.apache.org/ under the
"User Guide"
>> > >>>> link, but
>> > >>>> >>> there's nothing there anymore ...
>> > >>>> >>>
>> > >>>> >>> I need some guide on how to install Bigtop on
a physical
>> cluster
>> > >>>> ... It
>> > >>>> >>> was easy to create a virtual cluster on my machine
using
>> Vagrant,
>> > >>>> but I
>> > >>>> >>> cannot find a manual to install on a physical
cluster ...
>> > >>>> >>>
>> > >>>> >>> Cheers,
>> > >>>> >>> David
>> > >>>> >>>
>> > >>>>
>> > >>>
>> > >>>
>> > >>
>>
>

Mime
View raw message