Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 95B1818F0D for ; Wed, 9 Dec 2015 13:07:11 +0000 (UTC) Received: (qmail 8886 invoked by uid 500); 9 Dec 2015 13:07:11 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 8856 invoked by uid 500); 9 Dec 2015 13:07:11 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 8647 invoked by uid 99); 9 Dec 2015 13:07:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Dec 2015 13:07:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id E930A2C03DA for ; Wed, 9 Dec 2015 13:07:10 +0000 (UTC) Date: Wed, 9 Dec 2015 13:07:10 +0000 (UTC) From: "Andrew Onischuk (JIRA)" To: dev@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (AMBARI-14289) Rebalance HDFS fails with Operation not permitted error on an HA cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Andrew Onischuk created AMBARI-14289: ---------------------------------------- Summary: Rebalance HDFS fails with Operation not permitted err= or on an HA cluster Key: AMBARI-14289 URL: https://issues.apache.org/jira/browse/AMBARI-14289 Project: Ambari Issue Type: Bug Reporter: Andrew Onischuk Assignee: Andrew Onischuk Fix For: 2.2.0 Rebalance HDFS after enabling HA is failing with the below error: =20 =20 =20 { "href" : "http://172.22.107.111:8080/api/v1/clusters/cl1/requests/61/= tasks/929", "Tasks" : { "attempt_cnt" : 1, "cluster_name" : "cl1", "command" : "CUSTOM_COMMAND", "command_detail" : "REBALANCEHDFS HDFS/NAMENODE", "custom_command_name" : "REBALANCEHDFS", "end_time" : 1449539103294, "error_log" : "/var/lib/ambari-agent/data/errors-929.txt", "exit_code" : 1, "host_name" : "os-u14-dpaacu-ambari-hv-db-6-r-1.novalocal", "id" : 929, "output_log" : "/var/lib/ambari-agent/data/output-929.txt", "request_id" : 61, "role" : "NAMENODE", "stage_id" : 0, "start_time" : 1449539089090, "status" : "FAILED", "stderr" : "Traceback (most recent call last):\n File \"/var/lib/a= mbari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.p= y\", line 432, in \n NameNode().execute()\n File \"/usr/lib/pyt= hon2.6/site-packages/resource_management/libraries/script/script.py\", line= 217, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/com= mon-services/HDFS/2.1.0.2.0/package/scripts/namenode.py\", line 375, in reb= alancehdfs\n os.remove(ccache_file_path)\nOSError: [Errno 1] Operation n= ot permitted: '/tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8'", "stdout" : "Starting balancer with threshold =3D 10\n2015-12-08 01:= 44:58,099 - call['/usr/bin/klist -s /tmp/hdfs_rebalance_cc_6ec913166750834c= 9d9302d65b9c6cb8'] {'user': 'cstm-hdfs'}\n2015-12-08 01:44:58,140 - call re= turned (1, '######## Hortonworks #############\\nThis is MOTD message, adde= d for testing in qe infra')\n2015-12-08 01:44:58,141 - Execute['/usr/bin/ki= nit -c /tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8 -kt /etc/sec= urity/keytabs/hdfs.headless.keytab cstm-hdfs@EXAMPLE.COM'] {'user': 'cstm-h= dfs'}\nExecuting command ambari-sudo.sh su cstm-hdfs -l -s /bin/bash -c 'ex= port PATH=3D'\"'\"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/bin= :/usr/bin:/bin:/usr/local/games:/usr/games:/var/lib/ambari-agent:/usr/hdp/c= urrent/hadoop-client/bin'\"'\"' KRB5CCNAME=3D/tmp/hdfs_rebalance_cc_6ec9131= 66750834c9d9302d65b9c6cb8 ; hdfs --config /usr/hdp/current/hadoop-client/co= nf balancer -threshold 10'\n2015-12-08 01:44:58,182 - Execute['ambari-sudo.= sh su cstm-hdfs -l -s /bin/bash -c 'export PATH=3D'\"'\"'/usr/sbin:/sbin:/= usr/lib/ambari-server/*:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/= games:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'\"'\"' KRB5C= CNAME=3D/tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8 ; hdfs --co= nfig /usr/hdp/current/hadoop-client/conf balancer -threshold 10''] {'logout= put': False, 'on_new_line': handle_new_line}\n[balancer] ######## Hortonwor= ks #############\nThis is MOTD message, added for testing in qe infra\n[bal= ancer] 15/12/08 01:45:00 INFO balancer.Balancer: Using a threshold of 10.0\= n[balancer] 15/12/08 01:45:00 INFO balancer.Balancer: namenodes =3D [hdfs:= //nameservice]\n[balancer] 15/12/08 01:45:00 INFO balancer.Balancer: parame= ters =3D Balancer.BalancerParameters [BalancingPolicy.Node, threshold =3D 1= 0.0, max idle iteration =3D 5, #excluded nodes =3D 0, #included nodes =3D 0= , #source nodes =3D 0, #blockpools =3D 0, run during upgrade =3D false]\n[b= alancer] 15/12/08 01:45:00 INFO balancer.Balancer: included nodes =3D []\n[= balancer] 15/12/08 01:45:00 INFO balancer.Balancer: excluded nodes =3D []\n= [balancer] 15/12/08 01:45:00 INFO balancer.Balancer: source nodes =3D []\n[= balancer] Time Stamp Iteration# Bytes Already Moved Bytes L= eft To Move Bytes Being Moved[balancer] \n[balancer] 15/12/08 01:45:02 INF= O balancer.KeyManager: Block token params received from NN: update interval= =3D10hrs, 0sec, token lifetime=3D10hrs, 0sec\n[balancer] 15/12/08 01:45:02 = INFO block.BlockTokenSecretManager: Setting block keys\n[balancer] 15/12/08= 01:45:02 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0= sec\n[balancer] 15/12/08 01:45:02 INFO balancer.Balancer: dfs.balancer.move= dWinWidth =3D 5400000 (default=3D5400000)\n15/12/08 01:45:02 INFO balancer.= Balancer: dfs.balancer.moverThreads =3D 1000 (default=3D1000)\n15/12/08 01:= 45:02 INFO balancer.Balancer: dfs.balancer.dispatcherThreads =3D 200 (defau= lt=3D200)\n15/12/08 01:45:02 INFO balancer.Balancer: dfs.datanode.balance.m= ax.concurrent.moves =3D 5 (default=3D5)\n15/12/08 01:45:02 INFO balancer.Ba= lancer: dfs.balancer.getBlocks.size =3D 2147483648 (default=3D2147483648)\n= 15/12/08 01:45:02 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-= size =3D 10485760 (default=3D10485760)\n[balancer] 15/12/08 01:45:02 INFO b= lock.BlockTokenSecretManager: Setting block keys\n[balancer] 15/12/08 01:45= :02 INFO balancer.Balancer: dfs.balancer.max-size-to-move =3D 10737418240 (= default=3D10737418240)\n15/12/08 01:45:02 INFO balancer.Balancer: dfs.block= size =3D 134217728 (default=3D134217728)\n[balancer] 15/12/08 01:45:02 INFO= net.NetworkTopology: Adding a new node: /default-rack/172.22.107.103:1019\= n15/12/08 01:45:02 INFO net.NetworkTopology: Adding a new node: /default-ra= ck/172.22.107.100:1019\n15/12/08 01:45:02 INFO net.NetworkTopology: Adding = a new node: /default-rack/172.22.107.111:1019\n15/12/08 01:45:02 INFO balan= cer.Balancer: 0 over-utilized: []\n15/12/08 01:45:02 INFO balancer.Balancer= : 0 underutilized: []\nThe cluster is balanced. Exiting...\n[balancer] Dec = 8, 2015 1:45:02 AM 0 0 B 0 B = -1 B\n[balancer] Dec 8, 2015 1:45:02 AM Balancing took 3.459 = seconds", "structured_out" : { } } } =20 Please find the link to entire artifacts [here](http://linux- jenkins.qe.hortonworks.com/home/jenkins/qe-artifacts/os-u14-dpaacu-ambari-h= v- db-6-r/ambari-hv-db-1449547903/artifacts/screenshots/com.hw.ambari.ui.tests= .mo nitoring.admin_page.TestEnableHA/test11_rebalanceHDFSAfterEnablingHA/) -- This message was sent by Atlassian JIRA (v6.3.4#6332)