Return-Path: X-Original-To: apmail-bigtop-user-archive@www.apache.org Delivered-To: apmail-bigtop-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C95DCB83 for ; Fri, 28 Nov 2014 20:06:45 +0000 (UTC) Received: (qmail 40586 invoked by uid 500); 28 Nov 2014 20:06:44 -0000 Delivered-To: apmail-bigtop-user-archive@bigtop.apache.org Received: (qmail 40519 invoked by uid 500); 28 Nov 2014 20:06:44 -0000 Mailing-List: contact user-help@bigtop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@bigtop.apache.org Delivered-To: mailing list user@bigtop.apache.org Received: (qmail 40500 invoked by uid 99); 28 Nov 2014 20:06:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Nov 2014 20:06:44 +0000 X-ASF-Spam-Status: No, hits=-10.3 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS,USER_IN_DEF_SPF_WL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of prvs=402d680d9=leidle@amazon.com designates 207.171.184.29 as permitted sender) Received: from [207.171.184.29] (HELO smtp-fw-9102.amazon.com) (207.171.184.29) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Nov 2014 20:06:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1417205195; x=1448741195; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=7yhFlVFlPmk1GgbH2GqiuKATfPP6alZXqY8Kp1Bn1Wc=; b=bk/M7WE9Lzx8WEq53m/hAgf54cuWJhwdCCBrBhBRipcID/Kqpru7AN6B M7qhFJcH3CliBYGFh/rIwyS/U7lC/qvov8DjTWf8EqcfBsSw7cvXH1gZs xC+sUjEZe3BmTw4ggQ011kTHoDO17nT1p6LRvcATC9kogSzRMsNjKGIF+ Y=; X-IronPort-AV: E=Sophos;i="5.07,478,1413244800"; d="scan'208,217";a="170324623" Received: from email-inbound-relay-64002.pdx4.amazon.com ([10.220.169.156]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 28 Nov 2014 20:06:13 +0000 Received: from ex10-hub-9004.ant.amazon.com (pdx2-ws-svc-lb17-vlan2.amazon.com [10.247.140.66]) by email-inbound-relay-64002.pdx4.amazon.com (8.14.7/8.14.7) with ESMTP id sASK6DrZ011207 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=OK); Fri, 28 Nov 2014 20:06:13 GMT Received: from EX10-MBX-9003.ant.amazon.com ([fe80::b9e5:5388:f95f:c940]) by ex10-hub-9004.ant.amazon.com ([::1]) with mapi id 14.03.0181.006; Fri, 28 Nov 2014 12:06:12 -0800 From: "Leidle, Rob" To: "user@bigtop.apache.org" CC: "dev@bigtop.apache.org" Subject: Re: Problem using puppet scripts to configure bigtop on AmazonLinux Thread-Topic: Problem using puppet scripts to configure bigtop on AmazonLinux Thread-Index: AQHQCz+LSgtMW2sGkEyVizWUgVaJs5x2dv6A Date: Fri, 28 Nov 2014 20:06:11 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.6.141106 x-originating-ip: [10.184.49.66] Content-Type: multipart/alternative; boundary="_000_D09E159C36B8Bleidleamazoncom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_D09E159C36B8Bleidleamazoncom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable ++dev@bigtop.apache.org From: , "Leidle, Rob" > Reply-To: "user@bigtop.apache.org" > Date: Friday, November 28, 2014 at 11:14 AM To: "user@bigtop.apache.org" > Subject: Problem using puppet scripts to configure bigtop on AmazonLinux Hello all, I am trying to configure & install Bigtop 0.8.0 using the puppet= scripts on AmazonLinux on EC2. Thus far, almost everything has worked besi= des one minor change I have made to the site.pp manifest. However, I am run= ning into a problem, it seems that the services such as proxy server or nam= enode are not immediately starting. You can see the error below in the purp= le text related to namenode. info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-yarn-resourcemanager] info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-mapreduce-historyserver] info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-yarn-nodemanager] info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-yarn-proxyserver] debug: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml= ]: The container Class[Hadoop::Common-yarn] will propagate my refresh event debug: Service[hadoop-yarn-proxyserver](provider=3Dredhat): Executing '/sbi= n/service hadoop-yarn-proxyserver status' debug: Service[hadoop-yarn-proxyserver](provider=3Dredhat): Executing '/sbi= n/service hadoop-yarn-proxyserver start' err: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service= [hadoop-yarn-proxyserver]/ensure: change from stopped to running failed: Co= uld not start Service[hadoop-yarn-proxyserver]: Execution of '/sbin/service= hadoop-yarn-proxyserver start' returned 3: at /mnt/var/lib/bootstrap-acti= ons/1/bigtop-0.8.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:48= 3 debug: Service[hadoop-yarn-proxyserver](provider=3Dredhat): Executing '/sbi= n/service hadoop-yarn-proxyserver status' debug: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Servi= ce[hadoop-yarn-proxyserver]: Skipping restart; service is not running notice: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Serv= ice[hadoop-yarn-proxyserver]: Triggered 'refresh' from 4 events debug: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Servi= ce[hadoop-yarn-proxyserver]: The container Hadoop::Proxyserver[proxyserver]= will propagate my refresh event debug: Hadoop::Proxyserver[proxyserver]: The container Class[Hadoop_head_no= de] will propagate my refresh event debug: Class[Hadoop::Common-yarn]: The container Stage[main] will propagate= my refresh event debug: Service[hadoop-hdfs-namenode](provider=3Dredhat): Executing '/sbin/s= ervice hadoop-hdfs-namenode status' debug: Service[hadoop-hdfs-namenode](provider=3Dredhat): Executing '/sbin/s= ervice hadoop-hdfs-namenode start' err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoo= p-hdfs-namenode]/ensure: change from stopped to running failed: Could not s= tart Service[hadoop-hdfs-namenode]: Execution of '/sbin/service hadoop-hdfs= -namenode start' returned 3: at /mnt/var/lib/bootstrap-actions/1/bigtop-0.= 8.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:335 debug: Service[hadoop-hdfs-namenode](provider=3Dredhat): Executing '/sbin/s= ervice hadoop-hdfs-namenode status' debug: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[had= oop-hdfs-namenode]: Skipping restart; service is not running notice: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[ha= doop-hdfs-namenode]: Triggered 'refresh' from 4 events debug: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[had= oop-hdfs-namenode]: The container Hadoop::Namenode[namenode] will propagate= my refresh event debug: Hadoop::Namenode[namenode]: The container Class[Hadoop_head_node] wi= ll propagate my refresh event notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package[= hadoop-hdfs-datanode]: Dependency Service[hadoop-hdfs-namenode] has failure= s: true warning: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package= [hadoop-hdfs-datanode]: Skipping because of failed dependencies notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mn= t1/hdfs]: Dependency Service[hadoop-hdfs-namenode] has failures: true warning: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/m= nt1/hdfs]: Skipping because of failed dependencies notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mn= t/hdfs]: Dependency Service[hadoop-hdfs-namenode] has failures: true The interesting part is that if I query namenode status eventually the name= node will show up as started even though I have not taken any other actions= : [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED] [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED] [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status Hadoop namenode is running [ OK ] The same goes for proxy server. The problem here is that all other dependen= cies of namenode do not install (such as resource manager, etc). I am using= the latest release of AmazonLinux (2014.09) and this has puppet 2.7.25-1. = I am not sure what to do about this issue, has anyone else experienced some= thing like this? Should I just move to puppet 3.x and only try to install o= ut of the Bigtop trunk (0.9.0)? --_000_D09E159C36B8Bleidleamazoncom_ Content-Type: text/html; charset="us-ascii" Content-ID: <88474200B85C00499CFAEEB15651AF14@ant.amazon.com> Content-Transfer-Encoding: quoted-printable
++dev@bigtop.apache.org

From: <Leidle>, "Leidle,= Rob" <leidle@amazon.com&g= t;
Reply-To: "user@bigtop.apache.org" <user@bigtop.apache.org>
Date: Friday, November 28, 2014 at = 11:14 AM
To: "user@bigtop.apache.org" <user@bigtop.apache.org>
Subject: Problem using puppet scrip= ts to configure bigtop on AmazonLinux

Hello all, I am trying to configure & install Bigtop 0.8.0 using t= he puppet scripts on AmazonLinux on EC2. Thus far, almost everything has wo= rked besides one minor change I have made to the site.pp manifest. However,= I am running into a problem, it seems that the services such as proxy server or namenode are not immediately sta= rting. You can see the error below in the purple text related to namenode.<= /div>

info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-yarn-resourcemanager]

info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-mapreduce-historyserver]

info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-yarn-nodemanager]

info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]= : Scheduling refresh of Service[hadoop-yarn-proxyserver]

debug: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml= ]: The container Class[Hadoop::Common-yarn] will propagate my refresh event=

debug: Service[hadoop-yarn-proxyserver](provider=3Dredhat): Executing '/sbi= n/service hadoop-yarn-proxyserver status'

debug: Service[hadoop-yarn-proxyserver](provider=3Dredhat): Executing '/sbi= n/service hadoop-yarn-proxyserver start'

err: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service= [hadoop-yarn-proxyserver]/ensure: change from stopped to running failed: Co= uld not start Service[hadoop-yarn-proxyserver]: Execution of '/sbin/service= hadoop-yarn-proxyserver start' returned 3:  at /mnt/var/lib/bootstrap-actions/1/bigtop-0.8.0/bigtop-= deploy/puppet/modules/hadoop/manifests/init.pp:483

debug: Service[hadoop-yarn-proxyserver](provider=3Dredhat): Executing '/sbi= n/service hadoop-yarn-proxyserver status'

debug: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Servi= ce[hadoop-yarn-proxyserver]: Skipping restart; service is not running

notice: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Serv= ice[hadoop-yarn-proxyserver]: Triggered 'refresh' from 4 events

debug: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Servi= ce[hadoop-yarn-proxyserver]: The container Hadoop::Proxyserver[proxyserver]= will propagate my refresh event

debug: Hadoop::Proxyserver[proxyserver]: The container Class[Hadoop_head_no= de] will propagate my refresh event

debug: Class[Hadoop::Common-yarn]: The container Stage[main] will propagate= my refresh event

debug: Service[hadoop-hdfs-namenode](provider=3Dredhat): Executing '/sbin/s= ervice hadoop-hdfs-namenode status'

debug: Service[hadoop-hdfs-namenode](provider=3Dredhat): Executing '/sbin/s= ervice hadoop-hdfs-namenode start'

err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoo= p-hdfs-namenode]/ensure: change from stopped to running failed: Could not s= tart Service[hadoop-hdfs-namenode]: Execution of '/sbin/service hadoop-hdfs= -namenode start' returned 3:  at /mnt/var/lib/bootstrap-actions/1/bigtop-0.8.0/bigtop-deploy/puppet/modules= /hadoop/manifests/init.pp:335

debug: Service[hadoop-hdfs-namenode](provider=3Dredhat): Executing '/sbin/s= ervice hadoop-hdfs-namenode status'

debug: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[had= oop-hdfs-namenode]: Skipping restart; service is not running

notice: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[ha= doop-hdfs-namenode]: Triggered 'refresh' from 4 events

debug: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[had= oop-hdfs-namenode]: The container Hadoop::Namenode[namenode] will propagate= my refresh event

debug: Hadoop::Namenode[namenode]: The container Class[Hadoop_head_node] wi= ll propagate my refresh event

notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package[= hadoop-hdfs-datanode]: Dependency Service[hadoop-hdfs-namenode] has failure= s: true

warning: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package= [hadoop-hdfs-datanode]: Skipping because of failed dependencies

notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mn= t1/hdfs]: Dependency Service[hadoop-hdfs-namenode] has failures: true

warning: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/m= nt1/hdfs]: Skipping because of failed dependencies

notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mn= t/hdfs]: Dependency Service[hadoop-hdfs-namenode] has failures: true



The interesting part is that if I query namenode status eventually the= namenode will show up as started even though I have not taken any other ac= tions:

[hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status=

Hadoop namenode is not running             &n= bsp;               [FAILED]

[hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status=

Hadoop namenode is not running             &n= bsp;               [FAILED]

[hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status=

Hadoop namenode is running              =                   [  OK=   ]


The same goes for proxy server. The problem here is that all other dep= endencies of namenode do not install (such as resource manager, etc). I am = using the latest release of AmazonLinux (2014.09) and this has puppet 2.7.2= 5-1. I am not sure what to do about this issue, has anyone else experienced something like this? Should I just= move to puppet 3.x and only try to install out of the Bigtop trunk (0.9.0)= ?
--_000_D09E159C36B8Bleidleamazoncom_--