hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuqi Gu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-19387) HBase-spark snappy.SnappyError on Arm64
Date Thu, 30 Nov 2017 07:23:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yuqi Gu updated HBASE-19387:
----------------------------
    Priority: Minor  (was: Major)

> HBase-spark snappy.SnappyError on Arm64
> ---------------------------------------
>
>                 Key: HBASE-19387
>                 URL: https://issues.apache.org/jira/browse/HBASE-19387
>             Project: HBase
>          Issue Type: Bug
>          Components: spark, test
>    Affects Versions: 3.0.0
>            Reporter: Yuqi Gu
>            Priority: Minor
>
> When running the hbase-spark Unit tests on Arm64, the failures are shown as follows:
>  
> {code:java}
> scalatest-maven-plugin:1.0:test (test) @ hbase-spark ---
> Discovery starting.
> Discovery completed in 2 seconds, 837 milliseconds.
> Run starting. Expected test count is: 79
> HBaseDStreamFunctionsSuite:
> Formatting using clusterid: testClusterID
> - bulkput to test HBase client *** FAILED ***
>   java.lang.reflect.InvocationTargetException:
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
>   at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
>   at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
>   at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
>   at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
>   ...
>   Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY]
no native library is found for os.name=Linux and os.arch=aarch64
>   at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
>   at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
>   at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
>   at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
>   ...
>   Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library
is found for os.name=Linux and os.arch=aarch64
>   at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331)
>   at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171)
>   at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152)
>   at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
>   at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
>   ...
> Formatting using clusterid: testClusterID
> PartitionFilterSuite:
> *** RUN ABORTED ***
>   java.lang.reflect.InvocationTargetException:
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
>   at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
>   at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
>   at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
>   at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
>   ...
>   Cause: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not
initialize class org.xerial.snappy.Snappy
>   at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
>   at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
>   at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
>   at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
>   ...
>   Cause: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
>   at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
>   at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
>   at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
>   at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
>   at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> {code}
> Root Cause: 
> The spark version in hbase is 1.6.0.
> And the snappy version based on hbase-1.6.0 do not support the Arm64.
> Fix it by upgrading snappy-java to 1.1.4 in hbase-spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message