ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balázs Bence Sári (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (AMBARI-22160) hadooplzo package installation failed on devdeploys
Date Wed, 07 Feb 2018 14:37:00 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-22160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355510#comment-16355510
] 

Balázs Bence Sári edited comment on AMBARI-22160 at 2/7/18 2:36 PM:
--------------------------------------------------------------------

While looking at the code (I had a package-installation issue on the multi-mpack branch) I
found a few things I suspect could be an error:

{code}
    # repositoryFile is the truth
    # package_version should be made to the form W_X_Y_Z_nnnn
    package_version = default("repositoryFile/repoVersion", None)

    # TODO remove legacy checks
    if package_version is None:
      package_version = default("roleParams/package_version", None)

    # TODO remove legacy checks
    if package_version is None:
      package_version = default("hostLevelParams/package_version", None)

    package_version = None
    if (package_version is None or '-' not in package_version) and default('/repositoryFile',
None):
      self.load_available_packages()
      package_name = self.get_package_from_available(name, self.available_packages_in_repos)
      if package_name is None:
        raise Fail("Cannot match package for regexp name {0}. Available packages: {1}".format(name,
self.available_packages_in_repos))
      return package_name

    if package_version is not None:
      package_version = package_version.replace('.', package_delimiter).replace('-', package_delimiter)
{code}

First {{package_version}} is set both conditionally and unconditionally, then it is set to
{{None}}. Also after it is set to {{None}} it is checked for being {{None}}.

I suspect the {{package_version = None}} line is a mistake. However, when I removed it I run
into another installation issue (Ambari 2.6 / HDP 2.5, cluster with HDFS + YARN + TEZ). Then
I changed the code to this below which seemed to work.

{code}
    # repositoryFile is the truth
    # package_version should be made to the form W_X_Y_Z_nnnn
    package_version = default("repositoryFile/repoVersion", None)

    # THESE TWO CHECKS BELOW ARE REMOVED
    # TODO remove legacy checks
    # if package_version is None:
    # package_version = default("roleParams/package_version", None)

    # TODO remove legacy checks
    # if package_version is None:
    #  package_version = default("hostLevelParams/package_version", None)

    # REMOVE SUSPECTEDLY MISTAKEN LINE: package_version = None
    # THE REST STAYS THE SAME
    if (package_version is None or '-' not in package_version) and default('/repositoryFile',
None):
      self.load_available_packages()
      package_name = self.get_package_from_available(name, self.available_packages_in_repos)
      if package_name is None:
        raise Fail("Cannot match package for regexp name {0}. Available packages: {1}".format(name,
self.available_packages_in_repos))
      return package_name

    if package_version is not None:
      package_version = package_version.replace('.', package_delimiter).replace('-', package_delimiter)
{code}



was (Author: bsari):
While looking at the code (I had a package-installation issue on the multi-mpack branch) I
found a few things I suspect could be an error:

{code}
    # repositoryFile is the truth
    # package_version should be made to the form W_X_Y_Z_nnnn
    package_version = default("repositoryFile/repoVersion", None)

    # TODO remove legacy checks
    if package_version is None:
      package_version = default("roleParams/package_version", None)

    # TODO remove legacy checks
    if package_version is None:
      package_version = default("hostLevelParams/package_version", None)

    package_version = None
    if (package_version is None or '-' not in package_version) and default('/repositoryFile',
None):
      self.load_available_packages()
      package_name = self.get_package_from_available(name, self.available_packages_in_repos)
      if package_name is None:
        raise Fail("Cannot match package for regexp name {0}. Available packages: {1}".format(name,
self.available_packages_in_repos))
      return package_name

    if package_version is not None:
      package_version = package_version.replace('.', package_delimiter).replace('-', package_delimiter)
{code}

First **package_version** is set, then it is set to **None**.

> hadooplzo package installation failed on devdeploys
> ---------------------------------------------------
>
>                 Key: AMBARI-22160
>                 URL: https://issues.apache.org/jira/browse/AMBARI-22160
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Andrew Onischuk
>            Assignee: Andrew Onischuk
>            Priority: Major
>             Fix For: 2.6.0
>
>         Attachments: AMBARI-22160.patch, AMBARI-22160.patch, AMBARI-22160.patch, AMBARI-22160.patch
>
>
> Installation log:
>     
>     
>     
>     2017-10-04 14:21:45,371 - Package['hadooplzo-2-6-3-0-159'] {'retry_on_repo_unavailability':
True, 'retry_count': 5}
>     2017-10-04 14:21:45,395 - Skipping installation of existing package hadooplzo-2-6-3-0-159
>     2017-10-04 14:21:45,396 - Package['libhdfs0-2-6-3-0-159'] {'retry_on_repo_unavailability':
True, 'retry_count': 5}
>     2017-10-04 14:21:45,420 - Skipping installation of existing package libhdfs0-2-6-3-0-159
>     2017-10-04 14:21:45,424 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents':
True, 'group': 'root'}
>     2017-10-04 14:21:45,431 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'),
'owner': 'root', 'group': 'root', 'mode': 0644}
>     2017-10-04 14:21:45,432 - Writing File['/etc/security/limits.d/hdfs.conf'] because
it doesn't exist
>     2017-10-04 14:21:45,432 - Changing permission for /etc/security/limits.d/hdfs.conf
from 640 to 644
>     2017-10-04 14:21:45,432 - XmlConfig['hadoop-policy.xml'] {'owner': 'cstm-hdfs', 'group':
'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {},
'configurations': ...}
>     2017-10-04 14:21:45,444 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
>     2017-10-04 14:21:45,444 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml']
{'owner': 'cstm-hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
'UTF-8'}
>     2017-10-04 14:21:45,453 - Writing File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml']
because contents don't match
>     2017-10-04 14:21:45,454 - Changing owner for /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml
from 0 to cstm-hdfs
>     2017-10-04 14:21:45,454 - XmlConfig['ssl-client.xml'] {'owner': 'cstm-hdfs', 'group':
'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {},
'configurations': ...}
>     2017-10-04 14:21:45,463 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml
>     2017-10-04 14:21:45,464 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml']
{'owner': 'cstm-hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
'UTF-8'}
>     2017-10-04 14:21:45,470 - Writing File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml']
because it doesn't exist
>     2017-10-04 14:21:45,470 - Changing owner for /usr/hdp/current/hadoop-client/conf/ssl-client.xml
from 0 to cstm-hdfs
>     2017-10-04 14:21:45,470 - Changing group for /usr/hdp/current/hadoop-client/conf/ssl-client.xml
from 0 to hadoop
>     2017-10-04 14:21:45,471 - Directory['/usr/hdp/current/hadoop-client/conf/secure']
{'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
>     2017-10-04 14:21:45,471 - Creating directory Directory['/usr/hdp/current/hadoop-client/conf/secure']
since it doesn't exist.
>     2017-10-04 14:21:45,471 - Changing group for /usr/hdp/current/hadoop-client/conf/secure
from 0 to hadoop
>     2017-10-04 14:21:45,472 - XmlConfig['ssl-client.xml'] {'owner': 'cstm-hdfs', 'group':
'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes':
{}, 'configurations': ...}
>     2017-10-04 14:21:45,482 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
>     2017-10-04 14:21:45,482 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml']
{'owner': 'cstm-hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
'UTF-8'}
>     2017-10-04 14:21:45,489 - Writing File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml']
because it doesn't exist
>     2017-10-04 14:21:45,489 - Changing owner for /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
from 0 to cstm-hdfs
>     2017-10-04 14:21:45,489 - Changing group for /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
from 0 to hadoop
>     2017-10-04 14:21:45,490 - XmlConfig['ssl-server.xml'] {'owner': 'cstm-hdfs', 'group':
'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {},
'configurations': ...}
>     2017-10-04 14:21:45,499 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml
>     2017-10-04 14:21:45,499 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml']
{'owner': 'cstm-hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
'UTF-8'}
>     2017-10-04 14:21:45,506 - Writing File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml']
because it doesn't exist
>     2017-10-04 14:21:45,506 - Changing owner for /usr/hdp/current/hadoop-client/conf/ssl-server.xml
from 0 to cstm-hdfs
>     2017-10-04 14:21:45,507 - Changing group for /usr/hdp/current/hadoop-client/conf/ssl-server.xml
from 0 to hadoop
>     2017-10-04 14:21:45,507 - XmlConfig['hdfs-site.xml'] {'owner': 'cstm-hdfs', 'group':
'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {u'final':
{u'dfs.support.append': u'true', u'dfs.datanode.data.dir': u'true', u'dfs.namenode.http-address':
u'true', u'dfs.namenode.name.dir': u'true', u'dfs.webhdfs.enabled': u'true', u'dfs.datanode.failed.volumes.tolerated':
u'true'}}, 'configurations': ...}
>     2017-10-04 14:21:45,516 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
>     2017-10-04 14:21:45,517 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml']
{'owner': 'cstm-hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
'UTF-8'}
>     2017-10-04 14:21:45,564 - Writing File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml']
because contents don't match
>     2017-10-04 14:21:45,564 - Changing owner for /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
from 0 to cstm-hdfs
>     2017-10-04 14:21:45,565 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir':
'/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {u'final':
{u'fs.defaultFS': u'true'}}, 'owner': 'cstm-hdfs', 'configurations': ...}
>     2017-10-04 14:21:45,574 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml
>     2017-10-04 14:21:45,575 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml']
{'owner': 'cstm-hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding':
'UTF-8'}
>     2017-10-04 14:21:45,604 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content':
Template('slaves.j2'), 'owner': 'cstm-hdfs'}
>     2017-10-04 14:21:45,605 - Writing File['/usr/hdp/current/hadoop-client/conf/slaves']
because contents don't match
>     2017-10-04 14:21:45,605 - Changing owner for /usr/hdp/current/hadoop-client/conf/slaves
from 0 to cstm-hdfs
>     2017-10-04 14:21:45,606 - Package['liblzo2-2'] {'retry_on_repo_unavailability': True,
'retry_count': 5}
>     2017-10-04 14:21:45,630 - Installing package liblzo2-2 ('/usr/bin/apt-get -q -o Dpkg::Options::=--force-confdef
--allow-unauthenticated --assume-yes install liblzo2-2')
>     2017-10-04 14:21:48,997 - Package['hadooplzo-2-6-3-0'] {'retry_on_repo_unavailability':
True, 'retry_count': 5}
>     2017-10-04 14:21:49,021 - Installing package hadooplzo-2-6-3-0 ('/usr/bin/apt-get
-q -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install hadooplzo-2-6-3-0')
>     2017-10-04 14:21:50,632 - Execution of '/usr/bin/apt-get -q -o Dpkg::Options::=--force-confdef
--allow-unauthenticated --assume-yes install hadooplzo-2-6-3-0' returned 100. Reading package
lists...
>     Building dependency tree...
>     Reading state information...
>     W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/ambari-hdp-1.list:1
and /etc/apt/sources.list.d/hdp.list:2
>     W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/ambari-hdp-1.list:1
and /etc/apt/sources.list.d/hdp.list:2
>     W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/ambari-hdp-1.list:2
and /etc/apt/sources.list.d/hdp.list:3
>     W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/ambari-hdp-1.list:2
and /etc/apt/sources.list.d/hdp.list:3
>     E: Unable to locate package hadooplzo-2-6-3-0
>     2017-10-04 14:21:50,632 - Failed to install package hadooplzo-2-6-3-0. Executing
'/usr/bin/apt-get update -qq'
>     2017-10-04 14:21:52,716 - Retrying to install package hadooplzo-2-6-3-0 after 30
seconds
>     
> Why we trying to install package several times using different package
> version?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message