bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Starina <david.star...@gmail.com>
Subject User guide?
Date Thu, 17 Mar 2016 06:17:59 GMT
That's right, I'm using CentOS 7.

--David

On Thursday, 17 March 2016, Konstantin Boudnik <cos@apache.org
<javascript:_e(%7B%7D,'cvml','cos@apache.org');>> wrote:

> Hmm... that's plain nuts... Looks like latest ignite is pulling in some
> crap
> like osgi... OMG, WTF?
>
> Thanks for discovering this David! Somehow it never got stepped upon in my
> tests. Are you on centos7?
>
> Cos
>
> On Wed, Mar 16, 2016 at 11:39PM, David Starina wrote:
> > Hi,
> >
> > Roman, Evans, thank you for the tips - that helped a lot. Still have some
> > problems though, an error when running puppet apply ...
> >
> > This is a sample of errors I got:
> >
> >
> > *Debug: Executing '/bin/yum -d 0 -e 0 -y install
> > ignite-hadoop-service'Error: Could not update: Execution of '/bin/yum -d
> 0
> > -e 0 -y install ignite-hadoop-service' returned 1: Error: Package:
> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>  Requires:
> > osgi(org.apache.servicemix.bundles.spring-beans)Error: Package:
> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>  Requires:
> > osgi(org.apache.servicemix.bundles.spring-context-support)Error: Package:
> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>  Requires:
> > osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package:
> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>  Requires:
> > osgi(org.apache.servicemix.bundles.spring-core)*
> > .....
> >
> > Also, after that:
> >
> >
> >
> >
> >
> >
> > *Notice:
> >
> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
> > Dependency Package[ignite-hadoop] has failures: trueWarning:
> >
> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
> > Skipping because of failed dependenciesNotice:
> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency
> > Package[openjdk-7-jre-headless] has failures: trueWarning:
> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Skipping
> > because of failed dependenciesNotice:
> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
> > Dependency Package[openjdk-7-jre-headless] has failures: trueWarning:
> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
> Skipping
> > because of failed dependenciesNotice:
> >
> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/ignite-hadoop/conf/default-config.xml]:
> > Dependency Package[ignite-hadoop] has failures: true*
> > .....
> >
> >
> > This is the "site.yaml" I used ("hadoop" wasn't listed in components,
> but I
> > think it should be there?)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs:  -
> > /data/1hadoop_cluster_node::cluster_components:  - hadoop  -
> ignite_hadoop
> > - hue  - mahout  - yarn  - zookeeperbigtop::jdk_package_name:
> > "openjdk-7-jre-headless"bigtop::bigtop_repo_uri:
> > "http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
> > <http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64>"*
> >
> >
> >
> >
> > Any idea what I am doing wrong?
> >
> > --David
> >
> > On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <evansye@apache.org> wrote:
> >
> > > David
> > >
> > > Most of the time you need to run same command on all nodes. "pdsh"
> would
> > > be a good tool to use.
> > >
> > > 1. You need to have puppet agent installed on every node. yum -y
> install
> > > puppet will do the trick, or you need to download repo, or use
> > > bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's no deploy
> effort
> > > other than just install the package.
> > >
> > > 2. Masterless means each puppet agent takes commands and runs
> > > independently.  We DONT use puppet master, which is a centralized
> master
> > > daemon to dispatch works to all the registered puppet agent.
> > >
> > > 3. Next, you need to copy bigtop puppet recipes and bigtop hiera
> > > configuration to all nodes, or at least make them available for all
> nodes
> > > via nfs.
> > >
> > > 4. Finally, execute the puppet apply command on all nodes and the
> puppet
> > > recipes will automatically figure out it's role. There no need to
> deploy
> > > head node first and then slaves.
> > >
> > > Ask if you still have questions.
> > >
> > >
> > > Thanks
> > >
> > > Evans
> > > 2016年3月15日 下午4:10於 "David Starina" <david.starina@gmail.com>寫道:
> > >
> > >> Hi guys,
> > >>
> > >> me again. I haven't set up Puppet before, can you just point me in the
> > >> right direction? I know this is not a Puppet mailing list, but I bet
> you
> > >> have some tips for me :-)
> > >>
> > >> So what do I need to do? Install Puppet on every node? What does it
> mean
> > >> that the mode of Puppet is "*masterless"* - do I just have to do a
> basic
> > >> installation of Puppet and not care much about setting it up, and then
> > >> deploy bigtop on every node separately? And when setting the Hadoop
> head
> > >> node - does that mean that I need to deploy Bigtop on this node
> first, and
> > >> Bigtop will know this is the master node by the url or IP I provide
> to it?
> > >> And then deploy Bigtop via Puppet on all the other nodes and they will
> > >> connect to the head node (which is the NameNode, I suppose?).
> > >>
> > >> I can do this via trial and error, but if you guys can help me reduce
> the
> > >> "error" part, that will be great :) Sorry for such beginner
> questions, I
> > >> haven't used Puppet before ... The reason I'm asking is, I got access
> to a
> > >> "physical" (actually virtual, but it's the same) environment for a
> limited
> > >> amount of time (two weeks), and I need to start testing on it asap -
> and
> > >> since my day job takes a lot of my time, if I wanted to do this
> without
> > >> help it could take me two weeks just to set up the environment - but
> I need
> > >> to install it, which is not the central part of my project, and then
> run
> > >> some ML algorithms on a text corpus, which will be the main part of
> the
> > >> project ...
> > >>
> > >> --David
> > >>
> > >> On Thu, Mar 10, 2016 at 11:28 AM, David Starina <
> david.starina@gmail.com>
> > >> wrote:
> > >>
> > >>> Thanks for the info!
> > >>>
> > >>> --David
> > >>>
> > >>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas <
> jayunit100.apache@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> I think there is enough info on youtube that you can cobble
> together a
> > >>>> syllabus if you need structured learning.
> > >>>>
> > >>>> See cos/Romans Bigtop talks which are on YouTube and then my Bigtop
> > >>>> internals hackathon video.   That gives you both the high and low
> level
> > >>>> details to do whatever you want.
> > >>>>
> > >>>> > On Mar 8, 2016, at 1:11 PM, Konstantin Boudnik <cos@apache.org>
> > >>>> wrote:
> > >>>> >
> > >>>> > We also had this discussion to start producing "The Big Book
of
> Apache
> > >>>> > Bigtop" like HBase does, but never got into the materialization
> part
> > >>>> of that
> > >>>> > project.
> > >>>> >
> > >>>> > Cos
> > >>>> >
> > >>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans Ye wrote:
> > >>>> >> Hi David,
> > >>>> >>
> > >>>> >> If you just want to take Bigtop binaries to deploy a cluster,
you
> > >>>> can refer
> > >>>> >> to our puppet management recipes:
> > >>>> >>
> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet
> > >>>> >>
> > >>>> >> BTW bigtop provisioner is the best source to reference
for how
> bigtop
> > >>>> >> provision a cluster:
> > >>>> >>
> > >>>>
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide
> > >>>> >>
> > >>>> >> Please ask us if you find anything confusing.
> > >>>> >>
> > >>>> >>
> > >>>> >> Evans
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> 2016-03-08 0:48 GMT+08:00 David Starina <david.starina@gmail.com
> >:
> > >>>> >>
> > >>>> >>> Hi,
> > >>>> >>>
> > >>>> >>> Is there some version of Bigtop User Guide available?
I know it
> > >>>> used to be
> > >>>> >>> on the be on http://bigtop.apache.org/ under the "User
Guide"
> > >>>> link, but
> > >>>> >>> there's nothing there anymore ...
> > >>>> >>>
> > >>>> >>> I need some guide on how to install Bigtop on a physical
cluster
> > >>>> ... It
> > >>>> >>> was easy to create a virtual cluster on my machine
using
> Vagrant,
> > >>>> but I
> > >>>> >>> cannot find a manual to install on a physical cluster
...
> > >>>> >>>
> > >>>> >>> Cheers,
> > >>>> >>> David
> > >>>> >>>
> > >>>>
> > >>>
> > >>>
> > >>
>

Mime
View raw message