bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: User guide?
Date Thu, 17 Mar 2016 18:45:21 GMT
On Thu, Mar 17, 2016 at 12:46PM, David Starina wrote:
> Konstantin,
> 
> So I should be safer using CentOS 6?

No, this isn't what I am saying. Centos7 is fine, it seems to be an issue with
the Ignite's internal dependencies somehow... I will try to find some time
over the weekend to check it out.

> Also - was I right to include "hadoop" in site.yaml components? It's not
> listed in the basic site.yaml, but I am guessing it is not installed
> automatically?

If it gives you what you want - why not? ;)

Cos

> -David
> 
> On Thu, Mar 17, 2016 at 7:17 AM, David Starina <david.starina@gmail.com>
> wrote:
> 
> > That's right, I'm using CentOS 7.
> >
> > --David
> >
> >
> > On Thursday, 17 March 2016, Konstantin Boudnik <cos@apache.org> wrote:
> >
> >> Hmm... that's plain nuts... Looks like latest ignite is pulling in some
> >> crap
> >> like osgi... OMG, WTF?
> >>
> >> Thanks for discovering this David! Somehow it never got stepped upon in my
> >> tests. Are you on centos7?
> >>
> >> Cos
> >>
> >> On Wed, Mar 16, 2016 at 11:39PM, David Starina wrote:
> >> > Hi,
> >> >
> >> > Roman, Evans, thank you for the tips - that helped a lot. Still have
> >> some
> >> > problems though, an error when running puppet apply ...
> >> >
> >> > This is a sample of errors I got:
> >> >
> >> >
> >> > *Debug: Executing '/bin/yum -d 0 -e 0 -y install
> >> > ignite-hadoop-service'Error: Could not update: Execution of '/bin/yum
> >> -d 0
> >> > -e 0 -y install ignite-hadoop-service' returned 1: Error: Package:
> >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
> >>  Requires:
> >> > osgi(org.apache.servicemix.bundles.spring-beans)Error: Package:
> >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
> >>  Requires:
> >> > osgi(org.apache.servicemix.bundles.spring-context-support)Error:
> >> Package:
> >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
> >>  Requires:
> >> > osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package:
> >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
> >>  Requires:
> >> > osgi(org.apache.servicemix.bundles.spring-core)*
> >> > .....
> >> >
> >> > Also, after that:
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *Notice:
> >> >
> >> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
> >> > Dependency Package[ignite-hadoop] has failures: trueWarning:
> >> >
> >> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
> >> > Skipping because of failed dependenciesNotice:
> >> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency
> >> > Package[openjdk-7-jre-headless] has failures: trueWarning:
> >> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Skipping
> >> > because of failed dependenciesNotice:
> >> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
> >> > Dependency Package[openjdk-7-jre-headless] has failures: trueWarning:
> >> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
> >> Skipping
> >> > because of failed dependenciesNotice:
> >> >
> >> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/ignite-hadoop/conf/default-config.xml]:
> >> > Dependency Package[ignite-hadoop] has failures: true*
> >> > .....
> >> >
> >> >
> >> > This is the "site.yaml" I used ("hadoop" wasn't listed in components,
> >> but I
> >> > think it should be there?)
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs:  -
> >> > /data/1hadoop_cluster_node::cluster_components:  - hadoop  -
> >> ignite_hadoop
> >> > - hue  - mahout  - yarn  - zookeeperbigtop::jdk_package_name:
> >> > "openjdk-7-jre-headless"bigtop::bigtop_repo_uri:
> >> > "http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
> >> > <http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64>"*
> >> >
> >> >
> >> >
> >> >
> >> > Any idea what I am doing wrong?
> >> >
> >> > --David
> >> >
> >> > On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <evansye@apache.org> wrote:
> >> >
> >> > > David
> >> > >
> >> > > Most of the time you need to run same command on all nodes. "pdsh"
> >> would
> >> > > be a good tool to use.
> >> > >
> >> > > 1. You need to have puppet agent installed on every node. yum -y
> >> install
> >> > > puppet will do the trick, or you need to download repo, or use
> >> > > bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's no deploy
> >> effort
> >> > > other than just install the package.
> >> > >
> >> > > 2. Masterless means each puppet agent takes commands and runs
> >> > > independently.  We DONT use puppet master, which is a centralized
> >> master
> >> > > daemon to dispatch works to all the registered puppet agent.
> >> > >
> >> > > 3. Next, you need to copy bigtop puppet recipes and bigtop hiera
> >> > > configuration to all nodes, or at least make them available for all
> >> nodes
> >> > > via nfs.
> >> > >
> >> > > 4. Finally, execute the puppet apply command on all nodes and the
> >> puppet
> >> > > recipes will automatically figure out it's role. There no need to
> >> deploy
> >> > > head node first and then slaves.
> >> > >
> >> > > Ask if you still have questions.
> >> > >
> >> > >
> >> > > Thanks
> >> > >
> >> > > Evans
> >> > > 2016年3月15日 下午4:10於 "David Starina" <david.starina@gmail.com>寫道:
> >> > >
> >> > >> Hi guys,
> >> > >>
> >> > >> me again. I haven't set up Puppet before, can you just point me
in
> >> the
> >> > >> right direction? I know this is not a Puppet mailing list, but
I bet
> >> you
> >> > >> have some tips for me :-)
> >> > >>
> >> > >> So what do I need to do? Install Puppet on every node? What does
it
> >> mean
> >> > >> that the mode of Puppet is "*masterless"* - do I just have to
do a
> >> basic
> >> > >> installation of Puppet and not care much about setting it up,
and
> >> then
> >> > >> deploy bigtop on every node separately? And when setting the Hadoop
> >> head
> >> > >> node - does that mean that I need to deploy Bigtop on this node
> >> first, and
> >> > >> Bigtop will know this is the master node by the url or IP I provide
> >> to it?
> >> > >> And then deploy Bigtop via Puppet on all the other nodes and they
> >> will
> >> > >> connect to the head node (which is the NameNode, I suppose?).
> >> > >>
> >> > >> I can do this via trial and error, but if you guys can help me
> >> reduce the
> >> > >> "error" part, that will be great :) Sorry for such beginner
> >> questions, I
> >> > >> haven't used Puppet before ... The reason I'm asking is, I got
> >> access to a
> >> > >> "physical" (actually virtual, but it's the same) environment for
a
> >> limited
> >> > >> amount of time (two weeks), and I need to start testing on it
asap -
> >> and
> >> > >> since my day job takes a lot of my time, if I wanted to do this
> >> without
> >> > >> help it could take me two weeks just to set up the environment
- but
> >> I need
> >> > >> to install it, which is not the central part of my project, and
then
> >> run
> >> > >> some ML algorithms on a text corpus, which will be the main part
of
> >> the
> >> > >> project ...
> >> > >>
> >> > >> --David
> >> > >>
> >> > >> On Thu, Mar 10, 2016 at 11:28 AM, David Starina <
> >> david.starina@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> Thanks for the info!
> >> > >>>
> >> > >>> --David
> >> > >>>
> >> > >>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas <
> >> jayunit100.apache@gmail.com>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> I think there is enough info on youtube that you can cobble
> >> together a
> >> > >>>> syllabus if you need structured learning.
> >> > >>>>
> >> > >>>> See cos/Romans Bigtop talks which are on YouTube and then
my Bigtop
> >> > >>>> internals hackathon video.   That gives you both the high
and low
> >> level
> >> > >>>> details to do whatever you want.
> >> > >>>>
> >> > >>>> > On Mar 8, 2016, at 1:11 PM, Konstantin Boudnik <cos@apache.org>
> >> > >>>> wrote:
> >> > >>>> >
> >> > >>>> > We also had this discussion to start producing "The
Big Book of
> >> Apache
> >> > >>>> > Bigtop" like HBase does, but never got into the materialization
> >> part
> >> > >>>> of that
> >> > >>>> > project.
> >> > >>>> >
> >> > >>>> > Cos
> >> > >>>> >
> >> > >>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans Ye wrote:
> >> > >>>> >> Hi David,
> >> > >>>> >>
> >> > >>>> >> If you just want to take Bigtop binaries to deploy
a cluster,
> >> you
> >> > >>>> can refer
> >> > >>>> >> to our puppet management recipes:
> >> > >>>> >>
> >> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet
> >> > >>>> >>
> >> > >>>> >> BTW bigtop provisioner is the best source to
reference for how
> >> bigtop
> >> > >>>> >> provision a cluster:
> >> > >>>> >>
> >> > >>>>
> >> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide
> >> > >>>> >>
> >> > >>>> >> Please ask us if you find anything confusing.
> >> > >>>> >>
> >> > >>>> >>
> >> > >>>> >> Evans
> >> > >>>> >>
> >> > >>>> >>
> >> > >>>> >>
> >> > >>>> >> 2016-03-08 0:48 GMT+08:00 David Starina <
> >> david.starina@gmail.com>:
> >> > >>>> >>
> >> > >>>> >>> Hi,
> >> > >>>> >>>
> >> > >>>> >>> Is there some version of Bigtop User Guide
available? I know it
> >> > >>>> used to be
> >> > >>>> >>> on the be on http://bigtop.apache.org/ under
the "User Guide"
> >> > >>>> link, but
> >> > >>>> >>> there's nothing there anymore ...
> >> > >>>> >>>
> >> > >>>> >>> I need some guide on how to install Bigtop
on a physical
> >> cluster
> >> > >>>> ... It
> >> > >>>> >>> was easy to create a virtual cluster on my
machine using
> >> Vagrant,
> >> > >>>> but I
> >> > >>>> >>> cannot find a manual to install on a physical
cluster ...
> >> > >>>> >>>
> >> > >>>> >>> Cheers,
> >> > >>>> >>> David
> >> > >>>> >>>
> >> > >>>>
> >> > >>>
> >> > >>>
> >> > >>
> >>
> >

Mime
View raw message