Mailing-List: contact user-help@bigtop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@bigtop.apache.org
Date: Wed, 16 Mar 2016 17:03:44 -0700
From: Konstantin Boudnik <cos@apache.org>
To: user@bigtop.apache.org
Subject: Re: User guide?
Message-ID: <20160317000344.GS17583@tpx>
Mail-Followup-To: user@bigtop.apache.org
References: 
 <CANHbwQ=UVd0L1keMnE4TyU1TesXC1EcpbxLd92uFCciQwrkmdg@mail.gmail.com>
 <CAKGhRitEYEF39xB=ezG24gyiBWm24Z6MpWEpCMGdLQKJP6b9VQ@mail.gmail.com>
 <20160308181125.GA10860@tpx>
 <1B72D95D-FA30-4697-9A2E-9155B80D88B9@gmail.com>
 <CANHbwQnzpcVZrVyWe+2VQfaC6djYMBBsrWQ0W55XOOec4iJonQ@mail.gmail.com>
 <CANHbwQmmdbQEUbTPfiMHzFMEeSjwHcxn1iBNSi0n8kQFKtsR-g@mail.gmail.com>
 <CAKGhRiuR3CrPFvxiUdPuCHMR3NnzaQ-mnuwBNHo1oKQEXSrDrg@mail.gmail.com>
 <CANHbwQmTi9J9WntQ2_-xPqXSnTuKLkavyjVSvDmyvJo58Lx_bg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="htVlOAOs/VIaEhiC"
Content-Disposition: inline
In-Reply-To: 
 <CANHbwQmTi9J9WntQ2_-xPqXSnTuKLkavyjVSvDmyvJo58Lx_bg@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)


--htVlOAOs/VIaEhiC
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hmm... that's plain nuts... Looks like latest ignite is pulling in some crap
like osgi... OMG, WTF?=20

Thanks for discovering this David! Somehow it never got stepped upon in my
tests. Are you on centos7?

Cos

On Wed, Mar 16, 2016 at 11:39PM, David Starina wrote:
> Hi,
>=20
> Roman, Evans, thank you for the tips - that helped a lot. Still have some
> problems though, an error when running puppet apply ...
>=20
> This is a sample of errors I got:
>=20
>=20
> *Debug: Executing '/bin/yum -d 0 -e 0 -y install
> ignite-hadoop-service'Error: Could not update: Execution of '/bin/yum -d 0
> -e 0 -y install ignite-hadoop-service' returned 1: Error: Package:
> ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
> osgi(org.apache.servicemix.bundles.spring-beans)Error: Package:
> ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
> osgi(org.apache.servicemix.bundles.spring-context-support)Error: Package:
> ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
> osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package:
> ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)           Requires:
> osgi(org.apache.servicemix.bundles.spring-core)*
> .....
>=20
> Also, after that:
>=20
>=20
>=20
>=20
>=20
>=20
> *Notice:
> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-no=
de]/File[/etc/default/ignite-hadoop]:
> Dependency Package[ignite-hadoop] has failures: trueWarning:
> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-no=
de]/File[/etc/default/ignite-hadoop]:
> Skipping because of failed dependenciesNotice:
> /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency
> Package[openjdk-7-jre-headless] has failures: trueWarning:
> /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Skipping
> because of failed dependenciesNotice:
> /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
> Dependency Package[openjdk-7-jre-headless] has failures: trueWarning:
> /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]: Skippi=
ng
> because of failed dependenciesNotice:
> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-no=
de]/File[/etc/ignite-hadoop/conf/default-config.xml]:
> Dependency Package[ignite-hadoop] has failures: true*
> .....
>=20
>=20
> This is the "site.yaml" I used ("hadoop" wasn't listed in components, but=
 I
> think it should be there?)
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> *bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs:  -
> /data/1hadoop_cluster_node::cluster_components:  - hadoop  - ignite_hadoop
> - hue  - mahout  - yarn  - zookeeperbigtop::jdk_package_name:
> "openjdk-7-jre-headless"bigtop::bigtop_repo_uri:
> "http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
> <http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64>"*
>=20
>=20
>=20
>=20
> Any idea what I am doing wrong?
>=20
> --David
>=20
> On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <evansye@apache.org> wrote:
>=20
> > David
> >
> > Most of the time you need to run same command on all nodes. "pdsh" would
> > be a good tool to use.
> >
> > 1. You need to have puppet agent installed on every node. yum -y install
> > puppet will do the trick, or you need to download repo, or use
> > bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's no deploy ef=
fort
> > other than just install the package.
> >
> > 2. Masterless means each puppet agent takes commands and runs
> > independently.  We DONT use puppet master, which is a centralized master
> > daemon to dispatch works to all the registered puppet agent.
> >
> > 3. Next, you need to copy bigtop puppet recipes and bigtop hiera
> > configuration to all nodes, or at least make them available for all nod=
es
> > via nfs.
> >
> > 4. Finally, execute the puppet apply command on all nodes and the puppet
> > recipes will automatically figure out it's role. There no need to deploy
> > head node first and then slaves.
> >
> > Ask if you still have questions.
> >
> >
> > Thanks
> >
> > Evans
> > 2016=E5=B9=B43=E6=9C=8815=E6=97=A5 =E4=B8=8B=E5=8D=884:10=E6=96=BC "Dav=
id Starina" <david.starina@gmail.com>=E5=AF=AB=E9=81=93=EF=BC=9A
> >
> >> Hi guys,
> >>
> >> me again. I haven't set up Puppet before, can you just point me in the
> >> right direction? I know this is not a Puppet mailing list, but I bet y=
ou
> >> have some tips for me :-)
> >>
> >> So what do I need to do? Install Puppet on every node? What does it me=
an
> >> that the mode of Puppet is "*masterless"* - do I just have to do a bas=
ic
> >> installation of Puppet and not care much about setting it up, and then
> >> deploy bigtop on every node separately? And when setting the Hadoop he=
ad
> >> node - does that mean that I need to deploy Bigtop on this node first,=
 and
> >> Bigtop will know this is the master node by the url or IP I provide to=
 it?
> >> And then deploy Bigtop via Puppet on all the other nodes and they will
> >> connect to the head node (which is the NameNode, I suppose?).
> >>
> >> I can do this via trial and error, but if you guys can help me reduce =
the
> >> "error" part, that will be great :) Sorry for such beginner questions,=
 I
> >> haven't used Puppet before ... The reason I'm asking is, I got access =
to a
> >> "physical" (actually virtual, but it's the same) environment for a lim=
ited
> >> amount of time (two weeks), and I need to start testing on it asap - a=
nd
> >> since my day job takes a lot of my time, if I wanted to do this without
> >> help it could take me two weeks just to set up the environment - but I=
 need
> >> to install it, which is not the central part of my project, and then r=
un
> >> some ML algorithms on a text corpus, which will be the main part of the
> >> project ...
> >>
> >> --David
> >>
> >> On Thu, Mar 10, 2016 at 11:28 AM, David Starina <david.starina@gmail.c=
om>
> >> wrote:
> >>
> >>> Thanks for the info!
> >>>
> >>> --David
> >>>
> >>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas <jayunit100.apache@gmail.com>
> >>> wrote:
> >>>
> >>>> I think there is enough info on youtube that you can cobble together=
 a
> >>>> syllabus if you need structured learning.
> >>>>
> >>>> See cos/Romans Bigtop talks which are on YouTube and then my Bigtop
> >>>> internals hackathon video.   That gives you both the high and low le=
vel
> >>>> details to do whatever you want.
> >>>>
> >>>> > On Mar 8, 2016, at 1:11 PM, Konstantin Boudnik <cos@apache.org>
> >>>> wrote:
> >>>> >
> >>>> > We also had this discussion to start producing "The Big Book of Ap=
ache
> >>>> > Bigtop" like HBase does, but never got into the materialization pa=
rt
> >>>> of that
> >>>> > project.
> >>>> >
> >>>> > Cos
> >>>> >
> >>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans Ye wrote:
> >>>> >> Hi David,
> >>>> >>
> >>>> >> If you just want to take Bigtop binaries to deploy a cluster, you
> >>>> can refer
> >>>> >> to our puppet management recipes:
> >>>> >> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet
> >>>> >>
> >>>> >> BTW bigtop provisioner is the best source to reference for how bi=
gtop
> >>>> >> provision a cluster:
> >>>> >>
> >>>> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisione=
r+User+Guide
> >>>> >>
> >>>> >> Please ask us if you find anything confusing.
> >>>> >>
> >>>> >>
> >>>> >> Evans
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> 2016-03-08 0:48 GMT+08:00 David Starina <david.starina@gmail.com>:
> >>>> >>
> >>>> >>> Hi,
> >>>> >>>
> >>>> >>> Is there some version of Bigtop User Guide available? I know it
> >>>> used to be
> >>>> >>> on the be on http://bigtop.apache.org/ under the "User Guide"
> >>>> link, but
> >>>> >>> there's nothing there anymore ...
> >>>> >>>
> >>>> >>> I need some guide on how to install Bigtop on a physical cluster
> >>>> ... It
> >>>> >>> was easy to create a virtual cluster on my machine using Vagrant,
> >>>> but I
> >>>> >>> cannot find a manual to install on a physical cluster ...
> >>>> >>>
> >>>> >>> Cheers,
> >>>> >>> David
> >>>> >>>
> >>>>
> >>>
> >>>
> >>

--htVlOAOs/VIaEhiC
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iF4EAREIAAYFAlbp9GAACgkQenyFlstYjhLh6AEApF7mHhB4kKSRYhdqJxfGeMTg
YF7+YGJ/SQdfVbuHLdMA/1Snl0x4CMOoxtQRxU+oyDsnTDdhd38+61opC/Nrd1GY
=cRdm
-----END PGP SIGNATURE-----

--htVlOAOs/VIaEhiC--