ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AMBARI-14289) Rebalance HDFS fails with Operation not permitted error on an HA cluster
Date Wed, 09 Dec 2015 13:11:11 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-14289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Onischuk resolved AMBARI-14289.
--------------------------------------
    Resolution: Fixed

Committed to trunk and branch-2.2

> Rebalance HDFS fails with Operation not permitted error on an HA cluster
> ------------------------------------------------------------------------
>
>                 Key: AMBARI-14289
>                 URL: https://issues.apache.org/jira/browse/AMBARI-14289
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Andrew Onischuk
>            Assignee: Andrew Onischuk
>             Fix For: 2.2.0
>
>
> Rebalance HDFS after enabling HA is failing with the below error:
>     
>     
>     
>     {
>       "href" : "http://172.22.107.111:8080/api/v1/clusters/cl1/requests/61/tasks/929",
>       "Tasks" : {
>         "attempt_cnt" : 1,
>         "cluster_name" : "cl1",
>         "command" : "CUSTOM_COMMAND",
>         "command_detail" : "REBALANCEHDFS HDFS/NAMENODE",
>         "custom_command_name" : "REBALANCEHDFS",
>         "end_time" : 1449539103294,
>         "error_log" : "/var/lib/ambari-agent/data/errors-929.txt",
>         "exit_code" : 1,
>         "host_name" : "os-u14-dpaacu-ambari-hv-db-6-r-1.novalocal",
>         "id" : 929,
>         "output_log" : "/var/lib/ambari-agent/data/output-929.txt",
>         "request_id" : 61,
>         "role" : "NAMENODE",
>         "stage_id" : 0,
>         "start_time" : 1449539089090,
>         "status" : "FAILED",
>         "stderr" : "Traceback (most recent call last):\n  File \"/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py\",
line 432, in <module>\n    NameNode().execute()\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
line 217, in execute\n    method(env)\n  File \"/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py\",
line 375, in rebalancehdfs\n    os.remove(ccache_file_path)\nOSError: [Errno 1] Operation
not permitted: '/tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8'",
>         "stdout" : "Starting balancer with threshold = 10\n2015-12-08 01:44:58,099 -
call['/usr/bin/klist -s /tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8'] {'user':
'cstm-hdfs'}\n2015-12-08 01:44:58,140 - call returned (1, '######## Hortonworks #############\\nThis
is MOTD message, added for testing in qe infra')\n2015-12-08 01:44:58,141 - Execute['/usr/bin/kinit
-c /tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8 -kt /etc/security/keytabs/hdfs.headless.keytab
cstm-hdfs@EXAMPLE.COM'] {'user': 'cstm-hdfs'}\nExecuting command ambari-sudo.sh su cstm-hdfs
-l -s /bin/bash -c 'export  PATH='\"'\"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'\"'\"'
KRB5CCNAME=/tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8 ; hdfs --config /usr/hdp/current/hadoop-client/conf
balancer -threshold 10'\n2015-12-08 01:44:58,182 - Execute['ambari-sudo.sh su cstm-hdfs -l
-s /bin/bash -c 'export  PATH='\"'\"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'\"'\"'
KRB5CCNAME=/tmp/hdfs_rebalance_cc_6ec913166750834c9d9302d65b9c6cb8 ; hdfs --config /usr/hdp/current/hadoop-client/conf
balancer -threshold 10''] {'logoutput': False, 'on_new_line': handle_new_line}\n[balancer]
######## Hortonworks #############\nThis is MOTD message, added for testing in qe infra\n[balancer]
15/12/08 01:45:00 INFO balancer.Balancer: Using a threshold of 10.0\n[balancer] 15/12/08 01:45:00
INFO balancer.Balancer: namenodes  = [hdfs://nameservice]\n[balancer] 15/12/08 01:45:00 INFO
balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold
= 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes =
0, #blockpools = 0, run during upgrade = false]\n[balancer] 15/12/08 01:45:00 INFO balancer.Balancer:
included nodes = []\n[balancer] 15/12/08 01:45:00 INFO balancer.Balancer: excluded nodes =
[]\n[balancer] 15/12/08 01:45:00 INFO balancer.Balancer: source nodes = []\n[balancer] Time
Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved[balancer]
\n[balancer] 15/12/08 01:45:02 INFO balancer.KeyManager: Block token params received from
NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec\n[balancer] 15/12/08 01:45:02
INFO block.BlockTokenSecretManager: Setting block keys\n[balancer] 15/12/08 01:45:02 INFO
balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec\n[balancer] 15/12/08 01:45:02
INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)\n15/12/08 01:45:02
INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)\n15/12/08 01:45:02
INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)\n15/12/08 01:45:02
INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)\n15/12/08
01:45:02 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)\n15/12/08
01:45:02 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)\n[balancer]
15/12/08 01:45:02 INFO block.BlockTokenSecretManager: Setting block keys\n[balancer] 15/12/08
01:45:02 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)\n15/12/08
01:45:02 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)\n[balancer]
15/12/08 01:45:02 INFO net.NetworkTopology: Adding a new node: /default-rack/172.22.107.103:1019\n15/12/08
01:45:02 INFO net.NetworkTopology: Adding a new node: /default-rack/172.22.107.100:1019\n15/12/08
01:45:02 INFO net.NetworkTopology: Adding a new node: /default-rack/172.22.107.111:1019\n15/12/08
01:45:02 INFO balancer.Balancer: 0 over-utilized: []\n15/12/08 01:45:02 INFO balancer.Balancer:
0 underutilized: []\nThe cluster is balanced. Exiting...\n[balancer] Dec 8, 2015 1:45:02 AM
           0                  0 B                 0 B               -1 B\n[balancer] Dec 8,
2015 1:45:02 AM   Balancing took 3.459 seconds",
>         "structured_out" : { }
>       }
>     }
>     
> Please find the link to entire artifacts [here](http://linux-
> jenkins.qe.hortonworks.com/home/jenkins/qe-artifacts/os-u14-dpaacu-ambari-hv-
> db-6-r/ambari-hv-db-1449547903/artifacts/screenshots/com.hw.ambari.ui.tests.mo
> nitoring.admin_page.TestEnableHA/test11_rebalanceHDFSAfterEnablingHA/)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message