ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk" <aonis...@hortonworks.com>
Subject Review Request 31414: [Monarch] Cluster create failed with timeout on the client side
Date Wed, 25 Feb 2015 15:45:19 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31414/
-----------------------------------------------------------

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-9795
    https://issues.apache.org/jira/browse/AMBARI-9795


Repository: ambari


Description
-------

We saw one cluster create fail in Production with the below error during
Ambari install:

    
    
    
    Fail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP.list
-o APT::Get::List-Cleanup=0' returned 100. E: Could not get lock /var/lib/apt/lists/lock -
open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
    

Detailed log:

    
    
    
        {
          "href" : "http://headnode0.mjlinux15-ssh.c3.internal.cloudapp.net:8080/api/v1/clusters/mjlinux15/requests/1/tasks/27",
          "Tasks" : {
            "cluster_name" : "mjlinux15",
            "command_detail" : "DATANODE INSTALL",
            "id" : 27,
            "request_id" : 1,
            "stage_id" : 1,
            "status" : "FAILED",
            "stderr" : "2015-02-18 06:55:16,758 - Error while executing command 'install':\nTraceback
(most recent call last):\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
line 184, in execute\n    method(env)\n  File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py\",
line 33, in hook\n    install_repos()\n  File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\",
line 55, in install_repos\n    _alter_repo(\"create\", params.repo_info, template)\n  File
\"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\",
line 49, in _alter_repo\n    components = ubuntu_components, # ubuntu specific\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\",
line 148, in __init__\n    self.env.run()\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\",
line 151, in run\n
     self.run_action(resource, action)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\",
line 117, in run_action\n    provider_action()\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/repository.py\",
line 97, in action_create\n    retcode, out = checked_call(update_cmd_formatted, sudo=True)\n
 File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 67, in
inner\n    return function(command, **kwargs)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\",
line 79, in checked_call\n    return _call(command, logoutput, True, cwd, env, preexec_fn,
user, wait_for_finish, timeout, path, sudo, on_new_line)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\",
line 185, in _call\n    raise Fail(err_msg)\nFail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP.list
-o APT::Get::List-Cleanup=0' returned 100. E: Could not 
 get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable
to lock directory /var/lib/apt/lists/"
          }
        },
        {
          "href" : "http://headnode0.mjlinux15-ssh.c3.internal.cloudapp.net:8080/api/v1/clusters/mjlinux15/requests/1/tasks/28",
          "Tasks" : {
            "cluster_name" : "mjlinux15",
            "command_detail" : "METRIC_MONITOR INSTALL",
            "id" : 28,
            "request_id" : 1,
            "stage_id" : 1,
            "status" : "FAILED",
            "stderr" : "2015-02-18 06:55:18,118 - Error while executing command 'install':\nTraceback
(most recent call last):\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
line 184, in execute\n    method(env)\n  File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py\",
line 33, in hook\n    install_repos()\n  File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\",
line 55, in install_repos\n    _alter_repo(\"create\", params.repo_info, template)\n  File
\"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\",
line 49, in _alter_repo\n    components = ubuntu_components, # ubuntu specific\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\",
line 148, in __init__\n    self.env.run()\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\",
line 151, in run\n
     self.run_action(resource, action)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\",
line 117, in run_action\n    provider_action()\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/repository.py\",
line 97, in action_create\n    retcode, out = checked_call(update_cmd_formatted, sudo=True)\n
 File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 67, in
inner\n    return function(command, **kwargs)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\",
line 79, in checked_call\n    return _call(command, logoutput, True, cwd, env, preexec_fn,
user, wait_for_finish, timeout, path, sudo, on_new_line)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\",
line 185, in _call\n    raise Fail(err_msg)\nFail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP-UTILS.list
-o APT::Get::List-Cleanup=0' returned 100. E: Coul
 d not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE:
Unable to lock directory /var/lib/apt/lists/"
          }
        },
    

This is being seen on Azure with Ubuntu and we need to find out why this is
happening and see if this is an OS issue or Ambari issue.


Diffs
-----

  ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py bf24730 
  ambari-agent/src/main/python/ambari_agent/HostCleanup.py 398502e 
  ambari-agent/src/main/python/ambari_agent/HostInfo.py dc0834a 
  ambari-agent/src/test/python/ambari_agent/TestHostCheckReportFileHandler.py eb7efb2 
  ambari-agent/src/test/python/ambari_agent/TestHostCleanup.py f43784c 
  ambari-agent/src/test/python/ambari_agent/TestHostInfo.py c1b93d6 
  ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
888387c 
  ambari-server/src/main/resources/custom_actions/scripts/check_host.py ff2eabd 
  ambari-server/src/test/python/custom_actions/TestCheckHost.py 105a0b0 

Diff: https://reviews.apache.org/r/31414/diff/


Testing
-------

mvn clean test


Thanks,

Andrew Onischuk


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message