hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7433) java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
Date Fri, 03 Nov 2017 13:28:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated YARN-7433:
-----------------------------
    Environment:     (was: From centos6.5 upgrade centos7,    hadoop version(2.7.1) is compiled
on centos6.5 and Support snappy,   the copy runs on centos7.

but  yarn task error ,yarn  task errorlog: 
(hadoop Native Libraries Whether to recompile based on centos7???)
Error: java.lang.RuntimeException: native snappy library not available: this version of libhadoop
was built without snappy support.
	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.io.IOException: Spill failed)
    Description: 
>From centos6.5 upgrade centos7,    hadoop version(2.7.1) is compiled on centos6.5 and
Support snappy,   the copy runs on centos7.

but  yarn task error ,yarn  task errorlog: 
(hadoop Native Libraries Whether to recompile based on centos7???)
Error: java.lang.RuntimeException: native snappy library not available: this version of libhadoop
was built without snappy support.
	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.io.IOException: Spill failed

JIRA is for reporting bugs or tracking features rather than general user support.  The [mailing
lists|http://hadoop.apache.org/mailing_lists.html] would be more appropriate for this.

There are two possible reasons why this error is occurring.  Either the task cannot find libhadoop.so
or libhadoop.so was compiled without Snappy support.  For the former, see if the message "Unable
to load native-hadoop library for your platform" appears in the task logs.  If it does then
the task is not loading libhadoop.so and Snappy cannot work.  Check the LD_LIBRARY_PATH environment
variable setting to make sure the task can locate libhadoop.so.  If it is loading libhadoop.so
properly then the error means the native library was compiled without Snappy support and needs
to be rebuilt with -Drequire.snappy=true


> java.lang.RuntimeException: native snappy library not available: this version of libhadoop
was built without snappy support.
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-7433
>                 URL: https://issues.apache.org/jira/browse/YARN-7433
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: gehaijiang
>
> From centos6.5 upgrade centos7,    hadoop version(2.7.1) is compiled on centos6.5 and
Support snappy,   the copy runs on centos7.
> but  yarn task error ,yarn  task errorlog: 
> (hadoop Native Libraries Whether to recompile based on centos7???)
> Error: java.lang.RuntimeException: native snappy library not available: this version
of libhadoop was built without snappy support.
> 	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
> 	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
> 	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> 	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
> 	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
> 	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511)
> 	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Error: java.io.IOException: Spill failed



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message