hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
Date Wed, 06 Jan 2016 14:50:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085602#comment-15085602
] 

Jason Lowe commented on MAPREDUCE-6577:
---------------------------------------

Thanks, Sangjin!  Could you also commit this to branch-2.8? I find it odd that the fix is
important enough to backport to 2.6 and 2.7 but not important enough for 2.8.  Users updating
from those older releases to 2.8 will accidentally drop this fix.

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> --------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6577
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.6.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Critical
>             Fix For: 2.7.3, 2.9.0, 2.6.4
>
>         Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, MAPREDUCE-6577.03.patch,
MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured
to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to
load native-hadoop library for your platform... using builtin-java classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will fail. For
example, an uber-AM with lz4 compression for the mapper task will fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher:
Exception running local (uberized) 'child' : java.lang.RuntimeException: native lz4 library
not available
> 	at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> 	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> 	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> 	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
> 	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message