bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: Problem using puppet scripts to configure bigtop on AmazonLinux
Date Sat, 29 Nov 2014 03:08:42 GMT
On Sat, Nov 29, 2014 at 01:43AM, Leidle, Rob wrote:
> Yes, I ran into Bigtop-1522 and figured out I needed to add mapred-app.
> Sorry, I wrote what I said in the previous email incorrectly, yes,
> resource manager does not install because the depdendency namenode does
> not install correctly. I will look more closely at the service logs to see
> if I can figure out why it isn╧t starting. The error code of Ё3╡ indicates
> from the /etc/init.d/hadoop-hdfs-namenode script that this means it can╧t
> find the running process 5 seconds after starting it.

Yes, please look into the logs - might be something obvious missed. We are
running these recipes for a good 3+ years and they are fairly well tested.
Would be good to fix last bugs if any ;)

Cos

> 
> On 11/28/14, 4:14 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
> 
> >On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
> >> Hello all, I am trying to configure & install Bigtop 0.8.0 using the
> >>puppet scripts on AmazonLinux on EC2. Thus far, almost everything has
> >>worked besides one minor change I have made to the site.pp manifest.
> >>However, I am running into a problem, it seems that the services such as
> >>proxy server or namenode are not immediately starting. You can see the
> >>error below in the purple text related to namenode.
> >> 
> >> 
> >> info: 
> >>/Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]:
> >>Scheduling refresh of Service[hadoop-yarn-resourcemanager]
> >> 
> >> info: 
> >>/Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]:
> >>Scheduling refresh of Service[hadoop-mapreduce-historyserver]
> >> 
> >> info: 
> >>/Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]:
> >>Scheduling refresh of Service[hadoop-yarn-nodemanager]
> >> 
> >> info: 
> >>/Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]:
> >>Scheduling refresh of Service[hadoop-yarn-proxyserver]
> >> 
> >> debug: 
> >>/Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]:
> >>The container Class[Hadoop::Common-yarn] will propagate my refresh event
> >> 
> >> debug: Service[hadoop-yarn-proxyserver](provider=redhat): Executing
> >>'/sbin/service hadoop-yarn-proxyserver status'
> >> 
> >> debug: Service[hadoop-yarn-proxyserver](provider=redhat): Executing
> >>'/sbin/service hadoop-yarn-proxyserver start'
> >> 
> >> err: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[ha
> >>doop-yarn-proxyserver]/ensure: change from stopped to running failed:
> >>Could not start Service[hadoop-yarn-proxyserver]: Execution of
> >>'/sbin/service hadoop-yarn-proxyserver start' returned 3:  at
> >>/mnt/var/lib/bootstrap-actions/1/bigtop-0.8.0/bigtop-deploy/puppet/module
> >>s/hadoop/manifests/init.pp:483
> >> 
> >> debug: Service[hadoop-yarn-proxyserver](provider=redhat): Executing
> >>'/sbin/service hadoop-yarn-proxyserver status'
> >> 
> >> debug: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[ha
> >>doop-yarn-proxyserver]: Skipping restart; service is not running
> >> 
> >> notice: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[ha
> >>doop-yarn-proxyserver]: Triggered 'refresh' from 4 events
> >> 
> >> debug: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[ha
> >>doop-yarn-proxyserver]: The container Hadoop::Proxyserver[proxyserver]
> >>will propagate my refresh event
> >> 
> >> debug: Hadoop::Proxyserver[proxyserver]: The container
> >>Class[Hadoop_head_node] will propagate my refresh event
> >> 
> >> debug: Class[Hadoop::Common-yarn]: The container Stage[main] will
> >>propagate my refresh event
> >> 
> >> debug: Service[hadoop-hdfs-namenode](provider=redhat): Executing
> >>'/sbin/service hadoop-hdfs-namenode status'
> >> 
> >> debug: Service[hadoop-hdfs-namenode](provider=redhat): Executing
> >>'/sbin/service hadoop-hdfs-namenode start'
> >> 
> >> err: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-h
> >>dfs-namenode]/ensure: change from stopped to running failed: Could not
> >>start Service[hadoop-hdfs-namenode]: Execution of '/sbin/service
> >>hadoop-hdfs-namenode start' returned 3:  at
> >>/mnt/var/lib/bootstrap-actions/1/bigtop-0.8.0/bigtop-deploy/puppet/module
> >>s/hadoop/manifests/init.pp:335
> >> 
> >> debug: Service[hadoop-hdfs-namenode](provider=redhat): Executing
> >>'/sbin/service hadoop-hdfs-namenode status'
> >> 
> >> debug: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-h
> >>dfs-namenode]: Skipping restart; service is not running
> >> 
> >> notice: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-h
> >>dfs-namenode]: Triggered 'refresh' from 4 events
> >> 
> >> debug: 
> >>/Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-h
> >>dfs-namenode]: The container Hadoop::Namenode[namenode] will propagate
> >>my refresh event
> >> 
> >> debug: Hadoop::Namenode[namenode]: The container
> >>Class[Hadoop_head_node] will propagate my refresh event
> >> 
> >> notice: 
> >>/Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package[hadoop
> >>-hdfs-datanode]: Dependency Service[hadoop-hdfs-namenode] has failures:
> >>true
> >> 
> >> warning: 
> >>/Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package[hadoop
> >>-hdfs-datanode]: Skipping because of failed dependencies
> >> 
> >> notice: 
> >>/Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mnt1/hdf
> >>s]: Dependency Service[hadoop-hdfs-namenode] has failures: true
> >> 
> >> warning: 
> >>/Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mnt1/hdf
> >>s]: Skipping because of failed dependencies
> >> 
> >> notice: 
> >>/Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mnt/hdfs
> >>]: Dependency Service[hadoop-hdfs-namenode] has failures: true
> >> 
> >> 
> >> The interesting part is that if I query namenode status eventually the
> >>namenode will show up as started even though I have not taken any other
> >>actions:
> >> 
> >> 
> >> [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode
> >>status
> >> 
> >> Hadoop namenode is not running                             [FAILED]
> >> 
> >> [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode
> >>status
> >> 
> >> Hadoop namenode is not running                             [FAILED]
> >> 
> >> [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode
> >>status
> >> 
> >> Hadoop namenode is running                                 [  OK  ]
> >> 
> >> The same goes for proxy server. The problem here is that all other
> >> dependencies of namenode do not install (such as resource manager,
> >>etc). I
> >> am using the latest release of AmazonLinux (2014.09) and this has puppet
> >> 2.7.25-1. I am not sure what to do about this issue, has anyone else
> >> experienced something like this? Should I just move to puppet 3.x and
> >>only
> >> try to install out of the Bigtop trunk (0.9.0)?
> >
> >ResourceManager isn't a dependency of namenode - it's the other way
> >around.
> >It's hard to say what's going on with your system without looking into
> >particular daemon logs. I'd suggest you check them and investigate what's
> >the
> >trouble is. 
> >
> >Also, there's a small issue BIGTOP-1522 with nodemanager recipes if you're
> >installing a custom set of components, which might or not affect you
> >
> >Cos
> >
> 

Mime
View raw message