Mailing-List: contact ambari-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: ambari-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of smohanty@hortonworks.com
 designates 209.85.160.51 as permitted sender)
User-Agent: Microsoft-MacOutlook/14.2.5.121010
Date: Wed, 18 Sep 2013 07:47:41 -0700
Subject: Re: Problem installing Datanode
From: Sumit Mohanty <smohanty@hortonworks.com>
To: <ambari-dev@incubator.apache.org>
Message-ID: <CE5F0A15.2CE1C%smohanty@hortonworks.com>
Thread-Topic: Problem installing Datanode
In-Reply-To: <9930FD44-833A-49C1-BE11-2775B22F271B@hortonworks.com>
Mime-version: 1.0
Content-type: text/plain; charset=US-ASCII

Jon,

A task can timeout after 10 minutes. Datanode is typically the first
component to be installed. So it brings in the most number of new packages.
We have noticed such timeouts if the environment has slow connectivity. I
had investigated a similar situation with John few months back.

Typically, retry will fix it.

If it helps, you can setup a local repo where the installation does not
have to download packages from S3. What version of HDP and Ambari are you
using?

-Sumit

On 9/18/13 7:34 AM, "Jon Maron" <jmaron@hortonworks.com> wrote:

>Hi,
>
>  We are working with the savanna QA folks, and during attempt to create
>a cluster or scale a cluster they are seeing failures during service
>installation.  Below is an example of a DATANODE related failure.  It
>appears that the installation may be proceeding well but simply times
>out?  Is there any additional diagnoses we can do to ascertain the issue
>trigger the failure?
>
>Here is the JSON output for the failure:
>
>{
>  "href" : 
>"http://172.18.168.48:8080/api/v1/clusters/test-cluster/requests/1/tasks/7
>",
>  "Tasks" : {
>    "attempt_cnt" : 1,
>    "cluster_name" : "test-cluster",
>    "command" : "INSTALL",
>    "exit_code" : 999,
>    "host_name" : "test-cluster-worker-node-001.novalocal",
>    "id" : 7,
>    "request_id" : 1,
>    "role" : "DATANODE",
>    "stage_id" : 1,
>    "start_time" : 1379513216345,
>    "status" : "FAILED",
>    "stderr" : "none\n\n Puppet has been killed due to timeout",
>    "stdout" :notice:
>/Stage[main]/Hdp-repos::Process_repo/File[HDP]/ensure: defined content as
>'{md5}af147c3d39b51af76b25d4f95c69d1e9'
>notice: Finished catalog run in 0.04 seconds
>warning: Dynamic lookup of $service_state at
>/var/lib/ambari-agent/puppet/modules/hdp-hadoop/manifests/service.pp:76
>is deprecated.  Support will be removed in Puppet 2.8.  Use a
>fully-qualified variable name (e.g., $classname::variable) or
>parameterized classes.
>warning: Dynamic lookup of $service_state at
>/var/lib/ambari-agent/puppet/modules/hdp-hadoop/manifests/service.pp:85
>is deprecated.  Support will be removed in Puppet 2.8.  Use a
>fully-qualified variable name (e.g., $classname::variable) or
>parameterized classes.
>warning: Dynamic lookup of $configuration is deprecated.  Support will be
>removed in Puppet 2.8.  Use a fully-qualified variable name (e.g.,
>$classname::variable) or parameterized classes.
>warning: Dynamic lookup of $mapred-site is deprecated.  Support will be
>removed in Puppet 2.8.  Use a fully-qualified variable name (e.g.,
>$classname::variable) or parameterized classes.
>warning: Dynamic lookup of $tasktracker_port is deprecated.  Support will
>be removed in Puppet 2.8.  Use a fully-qualified variable name (e.g.,
>$classname::variable) or parameterized classes.
>warning: Dynamic lookup of $ambari_db_rca_url is deprecated.  Support
>will be removed in Puppet 2.8.  Use a fully-qualified variable name
>(e.g., $classname::variable) or parameterized classes.
>warning: Dynamic lookup of $ambari_db_rca_driver is deprecated.  Support
>will be removed in Puppet 2.8.  Use a fully-qualified variable name
>(e.g., $classname::variable) or parameterized classes.
>warning: Dynamic lookup of $ambari_db_rca_username is deprecated.
>Support will be removed in Puppet 2.8.  Use a fully-qualified variable
>name (e.g., $classname::variable) or parameterized classes.
>warning: Dynamic lookup of $ambari_db_rca_password is deprecated.
>Support will be removed in Puppet 2.8.  Use a fully-qualified variable
>name (e.g., $classname::variable) or parameterized classes.
>notice: /Stage[1]/Hdp::Iptables/Service[iptables]/ensure: ensure changed
>'running' to 'stopped'
>notice: /Stage[1]/Hdp::Create_smoke_user/File[/tmp/changeUid.sh]/ensure:
>defined content as '{md5}8118fe9ec0bca3d841d470ad02696adf'
>notice: 
>/Stage[1]/Hdp::Snappy::Package/Hdp::Package[snappy]/Hdp::Package::Process_
>pkg[snappy]/Package[snappy]/ensure: created
>notice: 
>/Stage[1]/Hdp/Hdp::Group[nagios_group]/Group[nagios_group]/ensure: created
>notice: /Stage[1]/Hdp/Hdp::User[nagios_user]/User[nagios]/ensure: created
>notice: 
>/Stage[1]/Hdp::Snmp/Hdp::Package[snmp]/Hdp::Package::Process_pkg[snmp]/Pac
>kage[net-snmp-utils]/ensure: created
>notice: 
>/Stage[1]/Hdp::Snmp/Hdp::Package[snmp]/Hdp::Package::Process_pkg[snmp]/Pac
>kage[net-snmp]/ensure: created
>notice: 
>/Stage[1]/Hdp::Snmp/Hdp::Package[snmp]/Hdp::Package::Process_pkg[snmp]/Hdp
>::Java::Package[snmp]/Exec[mkdir -p /tmp/HDP-artifacts/ ; curl -kf
>--retry 10 
>http://test-cluster-master-node-001.novalocal:8080/resources//jdk-6u31-lin
>ux-x64.bin -o /tmp/HDP-artifacts//jdk-6u31-linux-x64.bin snmp]/returns:
>executed successfully
>notice: 
>/Stage[1]/Hdp::Snmp/Hdp::Package[snmp]/Hdp::Package::Process_pkg[snmp]/Hdp
>::Java::Package[snmp]/Exec[mkdir -p /usr/jdk64 ; chmod +x
>/tmp/HDP-artifacts//jdk-6u31-linux-x64.bin; cd /usr/jdk64 ; echo A |
>/tmp/HDP-artifacts//jdk-6u31-linux-x64.bin -noregister > /dev/null 2>&1
>snmp]/returns: executed successfully
>notice: 
>/Stage[1]/Hdp::Snmp/Hdp::Package[snmp]/Hdp::Package::Process_pkg[snmp]/Hdp
>::Java::Package[snmp]/File[/usr/jdk64/jdk1.6.0_31/bin/java snmp]/ensure:
>created
>notice: 
>/Stage[1]/Hdp::Snmp/Hdp::Snmp-configfile[snmpd.conf]/Hdp::Configfile[/etc/
>snmp//snmpd.conf]/File[/etc/snmp//snmpd.conf]/content: content changed
>'{md5}8307434bc8ed4e2a7df4928fb4232778' to
>'{md5}f786955c0c36f7f5a4f375e3fe93c959'
>notice: /Stage[1]/Hdp::Snmp/Service[snmpd]/ensure: ensure changed
>'stopped' to 'running'
>notice: /Stage[1]/Hdp::Snmp/Service[snmpd]: Triggered 'refresh' from 1
>events
>notice: /Stage[1]/Hdp::Set_selinux/Hdp::Exec[/bin/echo 0 >
>/selinux/enforce]/Exec[/bin/echo 0 > /selinux/enforce]/returns: executed
>successfully
>notice: 
>/Stage[1]/Hdp::Snappy::Package/Hdp::Package[snappy]/Hdp::Package::Process_
>pkg[snappy]/Package[snappy-devel]/ensure: created
>notice: 
>/Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[64]/Hdp::Exec[hdp:
>:snappy::package::ln 64]/Exec[hdp::snappy::package::ln 64]/returns:
>executed successfully
>notice: 
>/Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp:
>:snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns:
>executed successfully
>notice: 
>/Stage[1]/Hdp/Hdp::Package[glibc]/Hdp::Package::Process_pkg[glibc]/Package
>[glibc.i686]/ensure: created
>notice: 
>/Stage[1]/Hdp/Hdp::Group[hdp_user_group]/Group[hdp_user_group]/ensure:
>created
>notice: 
>/Stage[1]/Hdp::Create_smoke_user/Hdp::User[smoke_user]/User[ambari_qa]/ens
>ure: created
>notice: /Stage[1]/Hdp::Create_smoke_user/Hdp::Exec[/tmp/changeUid.sh
>ambari_qa 1012 
>/tmp/hadoop-ambari_qa,/tmp/hsperfdata_ambari_qa,/home/ambari_qa,/tmp/ambar
>i_qa,/tmp/sqoop-ambari_qa 2>/dev/null]/Exec[/tmp/changeUid.sh ambari_qa
>1012 
>/tmp/hadoop-ambari_qa,/tmp/hsperfdata_ambari_qa,/home/ambari_qa,/tmp/ambar
>i_qa,/tmp/sqoop-ambari_qa 2>/dev/null]/returns: executed successfully
>err: 
>/Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop
>64]/Hdp::Package::Process_pkg[hadoop 64]/Package[hadoop-sbin]: Could not
>evaluate: Puppet::Util::Log requires a message
>
>  Any help would be appreciated
>
>-- Jon
>
>
>-- 
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity
>to 
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>that 
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>immediately 
>and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.