hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
Date Wed, 06 Jan 2016 23:10:40 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Junping Du updated MAPREDUCE-6577:
----------------------------------
    Target Version/s: 2.6.4

> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> --------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6577
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.6.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Critical
>             Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4
>
>         Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, MAPREDUCE-6577.03.patch,
MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured
to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to
load native-hadoop library for your platform... using builtin-java classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will fail. For
example, an uber-AM with lz4 compression for the mapper task will fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher:
Exception running local (uberized) 'child' : java.lang.RuntimeException: native lz4 library
not available
> 	at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> 	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> 	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> 	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
> 	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> 	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message