ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
Date Thu, 16 Nov 2017 14:47:01 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255431#comment-16255431
] 

Jonathan Hurley commented on AMBARI-22444:
------------------------------------------

Seems like this patch has issues with non-root and {{tarfile}}:

{code}
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py",
line 92, in _prepare_tez_tarball
    with closing(tarfile.open(tez_tarball_with_native_lib, "w:gz")) as new_tez_tarball:
  File "/usr/lib64/python2.6/tarfile.py", line 1670, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib64/python2.6/tarfile.py", line 1714, in gzopen
    fileobj = bltn_open(name, mode + "b")
IOError: [Errno 13] Permission denied: '/var/lib/ambari-agent/tmp/tez-native-tarball-staging/tez-native.tar.gz'
{code}

Attached [^AMBARI-22444-2.patch] to correct it

> Add Native Libraries To Tez Tarball
> -----------------------------------
>
>                 Key: AMBARI-22444
>                 URL: https://issues.apache.org/jira/browse/AMBARI-22444
>             Project: Ambari
>          Issue Type: Bug
>    Affects Versions: 2.6.1
>            Reporter: Jonathan Hurley
>            Assignee: Jonathan Hurley
>            Priority: Blocker
>             Fix For: 2.6.1
>
>         Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was
determined that the Tez tarball will need to have the hadoop native libraries added to it
so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries from the
Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec).
This is most likely due to missing native libraries for the codec.
> 	at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop libraries as well:
> {code}
> ├── tez
> │   ├── lib
> │   │   ├── native
> │   │   │   ├── libhadoop.a
> │   │   │   ├── libhadoop.so -> libhadoop.so.1.0.0
> │   │   │   ├── libhadoop.so.1.0.0
> │   │   │   ├── libhadooppipes.a
> │   │   │   ├── libhadooputils.a
> │   │   │   ├── libhdfs.a
> │   │   │   ├── libhdfs.so -> libhdfs.so.0.0.0
> │   │   │   ├── libhdfs.so.0.0.0
> │   │   │   ├── libsnappy.so -> libsnappy.so.1.1.4
> │   │   │   ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │   │   │   └── libsnappy.so.1.1.4
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message