Return-Path: 
 <mapreduce-issues-return-83158-apmail-hadoop-mapreduce-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2820A1836B
	for <apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org>;
 Thu, 17 Dec 2015 06:46:48 +0000 (UTC)
Received: (qmail 39455 invoked by uid 500); 17 Dec 2015 06:46:46 -0000
Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org
Received: (qmail 39342 invoked by uid 500); 17 Dec 2015 06:46:46 -0000
Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:mapreduce-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:mapreduce-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:mapreduce-issues@hadoop.apache.org>
List-Id: <mapreduce-issues.hadoop.apache.org>
Reply-To: mapreduce-issues@hadoop.apache.org
Delivered-To: mailing list mapreduce-issues@hadoop.apache.org
Received: (qmail 39155 invoked by uid 99); 17 Dec 2015 06:46:46 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Dec 2015 06:46:46 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id A1D832C1F6B
	for <mapreduce-issues@hadoop.apache.org>;
 Thu, 17 Dec 2015 06:46:46 +0000 (UTC)
Date: Thu, 17 Dec 2015 06:46:46 +0000 (UTC)
From: "Sangjin Lee (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12922612.1450334790000.50832.1450334806660@Atlassian.JIRA>
In-Reply-To: <JIRA.12922612.1450334790000@Atlassian.JIRA>
References: <JIRA.12922612.1450334790000@Atlassian.JIRA>
 <JIRA.12922612.1450334790517@arcas>
Subject: [jira] [Created] (MAPREDUCE-6577) MR AM unable to load native
 library without MR_AM_ADMIN_USER_ENV set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Sangjin Lee created MAPREDUCE-6577:
--------------------------------------

             Summary: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
                 Key: MAPREDUCE-6577
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mr-am
    Affects Versions: 2.6.0
            Reporter: Sangjin Lee
            Assignee: Sangjin Lee
            Priority: Critical


If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:

{noformat}
2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{noformat}

As a result, any code that needs the hadoop native library in the MR AM will fail. For example, an uber-AM with lz4 compression for the mapper task will fail:
{noformat}
2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.lang.RuntimeException: native lz4 library not available
	at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)