bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Starina <david.star...@gmail.com>
Subject Re: User guide?
Date Sat, 19 Mar 2016 16:39:29 GMT
Sorry guys,

Turn out I was leaving out JDK dependancy from site.yaml. So this went
through ok now. Thanks for all the help!

--David

On Sat, Mar 19, 2016 at 2:12 PM, David Starina <david.starina@gmail.com>
wrote:

> And this is not just about Ignite, I can't install anything else either ...
>
>
> --David
>
> On Sat, Mar 19, 2016 at 2:11 PM, David Starina <david.starina@gmail.com>
> wrote:
>
>> Guys, what am I (still) doing wrong?  I keep getting this error:
>> Error: Execution of '/bin/yum -d 0 -e 0 -y list jdk' returned 1: Error:
>> No matching Packages to list
>>
>> This probably means I am not installing Java in a way Bigtop is expecting
>> it. Do you have some idea what I should be doing differently? Of course,
>> installing Oracle JDK manually won't work, because it won't be listed in
>> yum. On the other hand, when I install OpenJDK via yum, "list jdk" doesn't
>> list any package either.
>>
>> What am I missing? Should I be adding some other repo to yum beforehand?
>>
>> --David
>>
>> On Thu, Mar 17, 2016 at 7:52 PM, David Starina <david.starina@gmail.com>
>> wrote:
>>
>>> Some more problems ... Tried this on CentOS 6.6 now ...
>>>
>>> 1) this line in site.yaml: "bigtop::jdk_package_name:
>>> "openjdk-7-jre-headless"" - not OK, you get the error:
>>>
>>>
>>>
>>> *Error: Execution of '/usr/bin/yum -d 0 -e 0 -y list
>>> openjdk-7-jre-headless' returned 1: Error: No matching Packages to
>>> listError: /Stage[main]/Main/Package[openjdk-7-jre-headless]/ensure: change
>>> from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y list
>>> openjdk-7-jre-headless' returned 1: Error: No matching Packages to list*
>>> 2) so, I installed OpenJDK manually ... Got the following error:
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Error: Could not start Service[ignite-hadoop-service]: Execution of
>>> '/sbin/service ignite-hadoop-service start' returned 1:
>>> ignite-hadoop-service: unrecognized serviceError:
>>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/Service[ignite-hadoop-service]/ensure:
>>> change from stopped to running failed: Could not start
>>> Service[ignite-hadoop-service]: Execution of '/sbin/service
>>> ignite-hadoop-service start' returned 1: ignite-hadoop-service:
>>> unrecognized serviceDebug: Executing '/bin/rpm -q jdk --nosignature
>>> --nodigest --qf %{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE}
>>> %{ARCH}\n'Debug: Executing '/usr/bin/yum -d 0 -e 0 -y list jdk'Error:
>>> Execution of '/usr/bin/yum -d 0 -e 0 -y list jdk' returned 1: Error: No
>>> matching Packages to listError: /Stage[main]/Main/Package[jdk]/ensure:
>>> change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0
>>> -y list jdk' returned 1: Error: No matching Packages to listNotice:
>>> /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency
>>> Package[jdk] has failures: true*
>>> .....
>>>
>>>
>>> So, '*/usr/bin/yum -d 0 -e 0 -y list jdk*' doesn't return anything ...
>>> this is how Java is listed when I run 'yum list installed':
>>> java-1.8.0-openjdk.x86_64
>>> java-1.8.0-openjdk-headless.x86_64
>>>
>>>
>>> So, still can't install Ignite for now ...
>>>
>>> --David
>>>
>>> On Thu, Mar 17, 2016 at 12:46 PM, David Starina <david.starina@gmail.com
>>> > wrote:
>>>
>>>> Konstantin,
>>>>
>>>> So I should be safer using CentOS 6?
>>>>
>>>> Also - was I right to include "hadoop" in site.yaml components? It's
>>>> not listed in the basic site.yaml, but I am guessing it is not installed
>>>> automatically?
>>>>
>>>> -David
>>>>
>>>> On Thu, Mar 17, 2016 at 7:17 AM, David Starina <david.starina@gmail.com
>>>> > wrote:
>>>>
>>>>> That's right, I'm using CentOS 7.
>>>>>
>>>>> --David
>>>>>
>>>>>
>>>>> On Thursday, 17 March 2016, Konstantin Boudnik <cos@apache.org>
wrote:
>>>>>
>>>>>> Hmm... that's plain nuts... Looks like latest ignite is pulling in
>>>>>> some crap
>>>>>> like osgi... OMG, WTF?
>>>>>>
>>>>>> Thanks for discovering this David! Somehow it never got stepped upon
>>>>>> in my
>>>>>> tests. Are you on centos7?
>>>>>>
>>>>>> Cos
>>>>>>
>>>>>> On Wed, Mar 16, 2016 at 11:39PM, David Starina wrote:
>>>>>> > Hi,
>>>>>> >
>>>>>> > Roman, Evans, thank you for the tips - that helped a lot. Still
>>>>>> have some
>>>>>> > problems though, an error when running puppet apply ...
>>>>>> >
>>>>>> > This is a sample of errors I got:
>>>>>> >
>>>>>> >
>>>>>> > *Debug: Executing '/bin/yum -d 0 -e 0 -y install
>>>>>> > ignite-hadoop-service'Error: Could not update: Execution of
>>>>>> '/bin/yum -d 0
>>>>>> > -e 0 -y install ignite-hadoop-service' returned 1: Error: Package:
>>>>>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>>>>>  Requires:
>>>>>> > osgi(org.apache.servicemix.bundles.spring-beans)Error: Package:
>>>>>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>>>>>  Requires:
>>>>>> > osgi(org.apache.servicemix.bundles.spring-context-support)Error:
>>>>>> Package:
>>>>>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>>>>>  Requires:
>>>>>> > osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package:
>>>>>> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop)
>>>>>>  Requires:
>>>>>> > osgi(org.apache.servicemix.bundles.spring-core)*
>>>>>> > .....
>>>>>> >
>>>>>> > Also, after that:
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > *Notice:
>>>>>> >
>>>>>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
>>>>>> > Dependency Package[ignite-hadoop] has failures: trueWarning:
>>>>>> >
>>>>>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]:
>>>>>> > Skipping because of failed dependenciesNotice:
>>>>>> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]:
>>>>>> Dependency
>>>>>> > Package[openjdk-7-jre-headless] has failures: trueWarning:
>>>>>> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]:
>>>>>> Skipping
>>>>>> > because of failed dependenciesNotice:
>>>>>> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
>>>>>> > Dependency Package[openjdk-7-jre-headless] has failures:
>>>>>> trueWarning:
>>>>>> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]:
>>>>>> Skipping
>>>>>> > because of failed dependenciesNotice:
>>>>>> >
>>>>>> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/ignite-hadoop/conf/default-config.xml]:
>>>>>> > Dependency Package[ignite-hadoop] has failures: true*
>>>>>> > .....
>>>>>> >
>>>>>> >
>>>>>> > This is the "site.yaml" I used ("hadoop" wasn't listed in
>>>>>> components, but I
>>>>>> > think it should be there?)
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > *bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs:
 -
>>>>>> > /data/1hadoop_cluster_node::cluster_components:  - hadoop  -
>>>>>> ignite_hadoop
>>>>>> > - hue  - mahout  - yarn  - zookeeperbigtop::jdk_package_name:
>>>>>> > "openjdk-7-jre-headless"bigtop::bigtop_repo_uri:
>>>>>> > "
>>>>>> http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
>>>>>> > <
>>>>>> http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64
>>>>>> >"*
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Any idea what I am doing wrong?
>>>>>> >
>>>>>> > --David
>>>>>> >
>>>>>> > On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <evansye@apache.org>
>>>>>> wrote:
>>>>>> >
>>>>>> > > David
>>>>>> > >
>>>>>> > > Most of the time you need to run same command on all nodes.
>>>>>> "pdsh" would
>>>>>> > > be a good tool to use.
>>>>>> > >
>>>>>> > > 1. You need to have puppet agent installed on every node.
yum -y
>>>>>> install
>>>>>> > > puppet will do the trick, or you need to download repo,
or use
>>>>>> > > bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's
no
>>>>>> deploy effort
>>>>>> > > other than just install the package.
>>>>>> > >
>>>>>> > > 2. Masterless means each puppet agent takes commands and
runs
>>>>>> > > independently.  We DONT use puppet master, which is a centralized
>>>>>> master
>>>>>> > > daemon to dispatch works to all the registered puppet agent.
>>>>>> > >
>>>>>> > > 3. Next, you need to copy bigtop puppet recipes and bigtop
hiera
>>>>>> > > configuration to all nodes, or at least make them available
for
>>>>>> all nodes
>>>>>> > > via nfs.
>>>>>> > >
>>>>>> > > 4. Finally, execute the puppet apply command on all nodes
and the
>>>>>> puppet
>>>>>> > > recipes will automatically figure out it's role. There
no need to
>>>>>> deploy
>>>>>> > > head node first and then slaves.
>>>>>> > >
>>>>>> > > Ask if you still have questions.
>>>>>> > >
>>>>>> > >
>>>>>> > > Thanks
>>>>>> > >
>>>>>> > > Evans
>>>>>> > > 2016年3月15日 下午4:10於 "David Starina" <david.starina@gmail.com>寫道:
>>>>>> > >
>>>>>> > >> Hi guys,
>>>>>> > >>
>>>>>> > >> me again. I haven't set up Puppet before, can you just
point me
>>>>>> in the
>>>>>> > >> right direction? I know this is not a Puppet mailing
list, but I
>>>>>> bet you
>>>>>> > >> have some tips for me :-)
>>>>>> > >>
>>>>>> > >> So what do I need to do? Install Puppet on every node?
What does
>>>>>> it mean
>>>>>> > >> that the mode of Puppet is "*masterless"* - do I just
have to do
>>>>>> a basic
>>>>>> > >> installation of Puppet and not care much about setting
it up,
>>>>>> and then
>>>>>> > >> deploy bigtop on every node separately? And when setting
the
>>>>>> Hadoop head
>>>>>> > >> node - does that mean that I need to deploy Bigtop
on this node
>>>>>> first, and
>>>>>> > >> Bigtop will know this is the master node by the url
or IP I
>>>>>> provide to it?
>>>>>> > >> And then deploy Bigtop via Puppet on all the other
nodes and
>>>>>> they will
>>>>>> > >> connect to the head node (which is the NameNode, I
suppose?).
>>>>>> > >>
>>>>>> > >> I can do this via trial and error, but if you guys
can help me
>>>>>> reduce the
>>>>>> > >> "error" part, that will be great :) Sorry for such
beginner
>>>>>> questions, I
>>>>>> > >> haven't used Puppet before ... The reason I'm asking
is, I got
>>>>>> access to a
>>>>>> > >> "physical" (actually virtual, but it's the same) environment
for
>>>>>> a limited
>>>>>> > >> amount of time (two weeks), and I need to start testing
on it
>>>>>> asap - and
>>>>>> > >> since my day job takes a lot of my time, if I wanted
to do this
>>>>>> without
>>>>>> > >> help it could take me two weeks just to set up the
environment -
>>>>>> but I need
>>>>>> > >> to install it, which is not the central part of my
project, and
>>>>>> then run
>>>>>> > >> some ML algorithms on a text corpus, which will be
the main part
>>>>>> of the
>>>>>> > >> project ...
>>>>>> > >>
>>>>>> > >> --David
>>>>>> > >>
>>>>>> > >> On Thu, Mar 10, 2016 at 11:28 AM, David Starina <
>>>>>> david.starina@gmail.com>
>>>>>> > >> wrote:
>>>>>> > >>
>>>>>> > >>> Thanks for the info!
>>>>>> > >>>
>>>>>> > >>> --David
>>>>>> > >>>
>>>>>> > >>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas <
>>>>>> jayunit100.apache@gmail.com>
>>>>>> > >>> wrote:
>>>>>> > >>>
>>>>>> > >>>> I think there is enough info on youtube that
you can cobble
>>>>>> together a
>>>>>> > >>>> syllabus if you need structured learning.
>>>>>> > >>>>
>>>>>> > >>>> See cos/Romans Bigtop talks which are on YouTube
and then my
>>>>>> Bigtop
>>>>>> > >>>> internals hackathon video.   That gives you
both the high and
>>>>>> low level
>>>>>> > >>>> details to do whatever you want.
>>>>>> > >>>>
>>>>>> > >>>> > On Mar 8, 2016, at 1:11 PM, Konstantin
Boudnik <
>>>>>> cos@apache.org>
>>>>>> > >>>> wrote:
>>>>>> > >>>> >
>>>>>> > >>>> > We also had this discussion to start producing
"The Big Book
>>>>>> of Apache
>>>>>> > >>>> > Bigtop" like HBase does, but never got
into the
>>>>>> materialization part
>>>>>> > >>>> of that
>>>>>> > >>>> > project.
>>>>>> > >>>> >
>>>>>> > >>>> > Cos
>>>>>> > >>>> >
>>>>>> > >>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans
Ye wrote:
>>>>>> > >>>> >> Hi David,
>>>>>> > >>>> >>
>>>>>> > >>>> >> If you just want to take Bigtop binaries
to deploy a
>>>>>> cluster, you
>>>>>> > >>>> can refer
>>>>>> > >>>> >> to our puppet management recipes:
>>>>>> > >>>> >>
>>>>>> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet
>>>>>> > >>>> >>
>>>>>> > >>>> >> BTW bigtop provisioner is the best
source to reference for
>>>>>> how bigtop
>>>>>> > >>>> >> provision a cluster:
>>>>>> > >>>> >>
>>>>>> > >>>>
>>>>>> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide
>>>>>> > >>>> >>
>>>>>> > >>>> >> Please ask us if you find anything
confusing.
>>>>>> > >>>> >>
>>>>>> > >>>> >>
>>>>>> > >>>> >> Evans
>>>>>> > >>>> >>
>>>>>> > >>>> >>
>>>>>> > >>>> >>
>>>>>> > >>>> >> 2016-03-08 0:48 GMT+08:00 David Starina
<
>>>>>> david.starina@gmail.com>:
>>>>>> > >>>> >>
>>>>>> > >>>> >>> Hi,
>>>>>> > >>>> >>>
>>>>>> > >>>> >>> Is there some version of Bigtop
User Guide available? I
>>>>>> know it
>>>>>> > >>>> used to be
>>>>>> > >>>> >>> on the be on http://bigtop.apache.org/
under the "User
>>>>>> Guide"
>>>>>> > >>>> link, but
>>>>>> > >>>> >>> there's nothing there anymore
...
>>>>>> > >>>> >>>
>>>>>> > >>>> >>> I need some guide on how to install
Bigtop on a physical
>>>>>> cluster
>>>>>> > >>>> ... It
>>>>>> > >>>> >>> was easy to create a virtual cluster
on my machine using
>>>>>> Vagrant,
>>>>>> > >>>> but I
>>>>>> > >>>> >>> cannot find a manual to install
on a physical cluster ...
>>>>>> > >>>> >>>
>>>>>> > >>>> >>> Cheers,
>>>>>> > >>>> >>> David
>>>>>> > >>>> >>>
>>>>>> > >>>>
>>>>>> > >>>
>>>>>> > >>>
>>>>>> > >>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message