Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2607F200CB6 for ; Thu, 29 Jun 2017 19:45:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 2577B160BED; Thu, 29 Jun 2017 17:45:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1CB0B160BC6 for ; Thu, 29 Jun 2017 19:45:10 +0200 (CEST) Received: (qmail 98277 invoked by uid 500); 29 Jun 2017 17:45:07 -0000 Mailing-List: contact issues-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list issues@ambari.apache.org Received: (qmail 98268 invoked by uid 99); 29 Jun 2017 17:45:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jun 2017 17:45:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E1B551806D5 for ; Thu, 29 Jun 2017 17:45:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.201 X-Spam-Level: X-Spam-Status: No, score=-99.201 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 04NSfzBIwl5B for ; Thu, 29 Jun 2017 17:45:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 1F34960CD9 for ; Thu, 29 Jun 2017 17:45:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id AA158E0DFC for ; Thu, 29 Jun 2017 17:45:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8A568245C9 for ; Thu, 29 Jun 2017 17:45:00 +0000 (UTC) Date: Thu, 29 Jun 2017 17:45:00 +0000 (UTC) From: "Andrew Onischuk (JIRA)" To: issues@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMBARI-21376) Using URI naming format in dfs.datanode.data.dir causing datanodes startup failure MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 29 Jun 2017 17:45:12 -0000 [ https://issues.apache.org/jira/browse/AMBARI-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-21376: ------------------------------------- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk and branch-2.5 > Using URI naming format in dfs.datanode.data.dir causing datanodes startup failure > ----------------------------------------------------------------------------------- > > Key: AMBARI-21376 > URL: https://issues.apache.org/jira/browse/AMBARI-21376 > Project: Ambari > Issue Type: Bug > Reporter: Andrew Onischuk > Assignee: Andrew Onischuk > Fix For: 2.5.0 > > Attachments: AMBARI-21376.patch > > > Using URI naming format in dfs.datanode.data.dir causing datanodes startup > failure. > I can replicate the issue locally @ master1.openstacklocal:8080/> > See task logs for starting DN for data1.openstacklocal / data2.openstacklocal > /data3.openstacklocal > > > > 2017-06-26 15:30:52,442 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf > 2017-06-26 15:30:52,603 - Stack Feature Version Info: stack_version=2.5, version=2.5.3.0-37, current_cluster_version=2.5.3.0-37 -> 2.5.3.0-37 > 2017-06-26 15:30:52,604 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf > User Group mapping (user_group) is missing in the hostLevelParams > 2017-06-26 15:30:52,605 - Group['livy'] {} > 2017-06-26 15:30:52,607 - Group['spark'] {} > 2017-06-26 15:30:52,607 - Group['hadoop'] {} > 2017-06-26 15:30:52,607 - Group['users'] {} > 2017-06-26 15:30:52,607 - Group['knox'] {} > 2017-06-26 15:30:52,608 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,609 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,609 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,610 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} > 2017-06-26 15:30:52,611 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,611 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,612 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} > 2017-06-26 15:30:52,612 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,613 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,614 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,614 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,615 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,616 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} > 2017-06-26 15:30:52,616 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} > 2017-06-26 15:30:52,618 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} > 2017-06-26 15:30:52,628 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if > 2017-06-26 15:30:52,628 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} > 2017-06-26 15:30:52,634 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} > 2017-06-26 15:30:52,635 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} > 2017-06-26 15:30:52,645 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if > 2017-06-26 15:30:52,646 - Group['hdfs'] {} > 2017-06-26 15:30:52,647 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']} > 2017-06-26 15:30:52,648 - FS Type: > 2017-06-26 15:30:52,649 - Directory['/etc/hadoop'] {'mode': 0755} > 2017-06-26 15:30:52,668 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'} > 2017-06-26 15:30:52,669 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} > 2017-06-26 15:30:52,687 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} > 2017-06-26 15:30:52,716 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'} > 2017-06-26 15:30:52,718 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'} > 2017-06-26 15:30:52,719 - Changing owner for /var/run/hadoop from 497 to root > 2017-06-26 15:30:52,719 - Changing group for /var/run/hadoop from 496 to root > 2017-06-26 15:30:52,719 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'} > 2017-06-26 15:30:52,725 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'} > 2017-06-26 15:30:52,728 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'} > 2017-06-26 15:30:52,733 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} > 2017-06-26 15:30:52,743 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'} > 2017-06-26 15:30:52,744 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} > 2017-06-26 15:30:52,745 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} > 2017-06-26 15:30:52,750 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} > 2017-06-26 15:30:52,759 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} > 2017-06-26 15:30:52,996 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf > 2017-06-26 15:30:53,002 - Stack Feature Version Info: stack_version=2.5, version=2.5.3.0-37, current_cluster_version=2.5.3.0-37 -> 2.5.3.0-37 > 2017-06-26 15:30:53,005 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf > 2017-06-26 15:30:53,011 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1} > 2017-06-26 15:30:53,067 - checked_call returned (0, '2.5.3.0-37', '') > 2017-06-26 15:30:53,080 - Execute['ambari-sudo.sh -H -E /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop datanode'] {'environment': {'HADOOP_SECURE_DN_USER': 'hdfs', 'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'only_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'} > 2017-06-26 15:30:58,159 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete']} > 2017-06-26 15:30:58,161 - checked_call['/usr/hdp/current/hadoop-hdfs-datanode/bin/hdfs dfsadmin -fs hdfs://siyona -D ipc.client.connect.max.retries=5 -D ipc.client.connect.retry.interval=1000 -getDatanodeInfo 0.0.0.0:8010'] {'tries': 1, 'user': 'hdfs'} > 2017-06-26 15:31:04,724 - DataNode has successfully shutdown. > 2017-06-26 15:31:04,726 - Pid file /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid is empty or does not exist > 2017-06-26 15:31:04,731 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} > 2017-06-26 15:31:04,740 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} > 2017-06-26 15:31:04,741 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} > 2017-06-26 15:31:04,752 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml > 2017-06-26 15:31:04,770 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} > 2017-06-26 15:31:04,789 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} > 2017-06-26 15:31:04,798 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml > 2017-06-26 15:31:04,798 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} > 2017-06-26 15:31:04,805 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'} > 2017-06-26 15:31:04,806 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...} > 2017-06-26 15:31:04,815 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml > 2017-06-26 15:31:04,815 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} > 2017-06-26 15:31:04,822 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} > 2017-06-26 15:31:04,830 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml > 2017-06-26 15:31:04,830 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} > 2017-06-26 15:31:04,838 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {'final': {'dfs.datanode.failed.volumes.tolerated': 'true', 'dfs.datanode.data.dir': 'true', 'dfs.namenode.name.dir': 'true', 'dfs.support.append': 'true', 'dfs.webhdfs.enabled': 'true'}}, 'configurations': ...} > 2017-06-26 15:31:04,847 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml > 2017-06-26 15:31:04,847 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} > 2017-06-26 15:31:04,926 - Writing File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] because contents don't match > 2017-06-26 15:31:04,927 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'fs.defaultFS': 'true'}}, 'owner': 'hdfs', 'configurations': ...} > 2017-06-26 15:31:04,936 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml > 2017-06-26 15:31:04,936 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} > 2017-06-26 15:31:04,968 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'root'} > 2017-06-26 15:31:04,970 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'group': 'hadoop', 'mode': 0751} > 2017-06-26 15:31:04,970 - Directory['/var/lib/ambari-agent/data/datanode'] {'create_parents': True, 'mode': 0755} > 2017-06-26 15:31:04,977 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/proc/sys/fs/binfmt_misc']. > 2017-06-26 15:31:04,977 - Mount point for directory file:///hadoop/hdfs/data is None > 2017-06-26 15:31:04,977 - Mount point for directory file:///hadoop1/hdfs/data is None > 2017-06-26 15:31:04,977 - Mount point for directory file:///hadoop/hdfs/data is None > 2017-06-26 15:31:04,978 - Forcefully ensuring existence and permissions of the directory: file:///hadoop/hdfs/data > 2017-06-26 15:31:04,978 - Directory['file:///hadoop/hdfs/data'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'ignore_failures': True, 'mode': 0755, 'owner': 'hdfs'} > > Command failed after 1 tries > -- This message was sent by Atlassian JIRA (v6.4.14#64029)