hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@apache.org>
Subject Re: Error : java.lang.UnsatisfiedLinkError: failed to load the required native library for netty
Date Tue, 03 Oct 2017 18:48:18 GMT
the linux box roughly as capable as your laptop?

On Tue, Oct 3, 2017 at 12:51 PM, Stack <stack@duboce.net> wrote:
> Tests passed eventually for me:
>
> real 47m6.498s
> user 5m29.671s
> sys 0m41.885s
>
> ... which is a big diff from macosx run.
>
> Need to look into this.
>
> St.Ack
>
>
> On Tue, Oct 3, 2017 at 10:03 AM, Stack <stack@duboce.net> wrote:
>
>> The below gets us further but now I see that the spark tests take a really
>> long time to run on linux but complete promptly on macosx (2m 55s).
>> Looking....
>>
>> St.Ack
>>
>> On Tue, Oct 3, 2017 at 9:13 AM, Stack <stack@duboce.net> wrote:
>>
>>> This seems to work for me. Does it work for you?
>>>
>>>
>>> diff --git a/hbase-spark/pom.xml b/hbase-spark/pom.xml
>>> index 594aa2a..6d191e3 100644
>>> --- a/hbase-spark/pom.xml
>>> +++ b/hbase-spark/pom.xml
>>> @@ -568,6 +568,9 @@
>>>            <junitxml>.</junitxml>
>>>            <filereports>WDF TestSuite.txt</filereports>
>>>            <parallel>false</parallel>
>>> +          <systemProperties>
>>> +            <org.apache.hadoop.hbase.shaded.io.netty.packagePrefix>org.
>>> apache.hadoop.hbase.shaded.</org.apache.hadoop.hbase.
>>> shaded.io.netty.packagePrefix>
>>> +          </systemProperties>
>>>          </configuration>
>>>          <executions>
>>>            <execution>
>>>
>>> St.Ack
>>>
>>> On Tue, Oct 3, 2017 at 8:45 AM, Amit Kabra <amitkabraiiit@gmail.com>
>>> wrote:
>>>
>>>> Thanks Stack / Sean Busbey for replying.
>>>>
>>>> OS : Ubuntu 16.04.2 , 64 bit.
>>>> Eclipse : Version: Neon.3 Release (4.6.3)
>>>> HBase branch : branch-2
>>>> Command line test to reproduce : mvn clean package
>>>> -Dtest=TestIncrementalBackup
>>>> Reproduce from eclipse , right click on TestIncBackupRestore and run as
>>>> junit from test class TestIncrementalBackup.
>>>> No I am not embedding hbase in my application. I have just checked out
>>>> hbase , switched to branch-2 and run the unit test from command line or
>>>> from eclipse. Failing with same error in both cases.
>>>> Yes the trailing period is also present.
>>>>
>>>> Thanks,
>>>> Amit Kabra.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Oct 3, 2017 at 8:53 PM, Stack <stack@duboce.net> wrote:
>>>>
>>>> > Thank you for the detail.
>>>> >
>>>> > Pardon the questions below asking for yet more detail. I am unable to
>>>> > reproduce locally or on another os (though we see this issue up on our
>>>> > build box).
>>>> >
>>>> > What is your OS when you see the below?
>>>> >
>>>> > On Tue, Oct 3, 2017 at 2:06 AM, Amit Kabra <amitkabraiiit@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hello,
>>>> > >
>>>> > > I am using "branch-2" branch of hbase, when I run unit test I get
>>>> > following
>>>> > > error for netty "java.lang.UnsatisfiedLinkError: failed to load
the
>>>> > > required native library"
>>>> > >
>>>> > >
>>>> > > This is running a unit test in your eclipse environment?
>>>> >
>>>> > You are trying to run an hbase-spark unit test when you see the above?
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > > *I already have following set in "maven-surefire-plugin" in pom.xml
>>>> as
>>>> > > per http://hbase.apache.org/book.html#thirdparty
>>>> > > <http://hbase.apache.org/book.html#thirdparty>*
>>>> > >
>>>> > >
>>>> > >
>>>> >
>>>> > Are you embedding hbase into your application?
>>>> >
>>>> >
>>>> >
>>>> > >             <systemPropertyVariables>
>>>> > >                 <!--
>>>> > >               <test.build.classes>${test.build.classes}</test.build.
>>>> > classe
>>>> > > s>
>>>> > >                 -->
>>>> > >               <!--For shaded netty, to find the relocated .so.
>>>> > >                    Trick from
>>>> > >                 https://stackoverflow.com/questions/33825743/rename-
>>>> > > files-inside-a-jar-using-some-maven-plugin
>>>> > > <https://stackoverflow.com/questions/33825743/rename-
>>>> > files-inside-a-jar-using-some-maven-plugin>
>>>> > >
>>>> > >                 The netty jar has a .so in it. Shading requires
>>>> rename of
>>>> > > the .so and then passing a system
>>>> > >                 property so netty finds the renamed .so and
>>>> associates it
>>>> > > w/ the relocated netty files.
>>>> > >
>>>> > >                 The relocated netty is in hbase-thirdparty
>>>> dependency.
>>>> > Just
>>>> > > set this propery globally rather
>>>> > >                 than per module.
>>>> > >                -->
>>>> > >               <org.apache.hadoop.hbase.shad
>>>> ed.io.netty.packagePrefix>
>>>> > > org.apache.hadoop.hbase.shaded.</org.apache.hadoop.
>>>> > hbase.shaded.io.netty.
>>>> > > packagePrefix>
>>>> > >             </systemPropertyVariables>
>>>> > >
>>>> > >
>>>> > >
>>>> > > *And I see in the code as per HBASE-18271, all io.netty is already
>>>> > replaced
>>>> > > with org.apache.hadoop.hbase.shaded.io.netty*
>>>> > >
>>>> > >
>>>> > The trailing period is also present?
>>>> >
>>>> >
>>>> >
>>>> > >
>>>> > > If I run a test from eclipse , I see the error immediately and
my
>>>> test
>>>> > > doesn't run, but when I run from command line , the test runs but
I
>>>> get
>>>> > the
>>>> > > error at the end when the mvn command finishes.
>>>> > >
>>>> > >
>>>> > > Is it any eclipse test?
>>>> >
>>>> > Thank you. Let me try and fix this this morning.
>>>> >
>>>> > S
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > > *Here is the complete error output.*
>>>> > >
>>>> > >
>>>> > >
>>>> > > [INFO]
>>>> > > [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @
>>>> > hbase-spark
>>>> > > ---
>>>> > > [INFO]
>>>> > > [INFO] --- scalatest-maven-plugin:1.0:test (test) @ hbase-spark
---
>>>> > > Discovery starting.
>>>> > > Discovery completed in 1 second, 558 milliseconds.
>>>> > > Run starting. Expected test count is: 79
>>>> > > HBaseDStreamFunctionsSuite:
>>>> > > Formatting using clusterid: testClusterID
>>>> > > *** RUN ABORTED ***
>>>> > >   java.io.IOException: Shutting down
>>>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.init(
>>>> > > MiniHBaseCluster.java:232)
>>>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(
>>>> > > MiniHBaseCluster.java:94)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCl
>>>> uster(
>>>> > > HBaseTestingUtility.java:1124)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:1078)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:949)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:943)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:872)
>>>> > >   at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.bef
>>>> oreAll(
>>>> > > HBaseDStreamFunctionsSuite.scala:41)
>>>> > >   at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAft
>>>> erAll.
>>>> > > scala:187)
>>>> > >   at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.bef
>>>> oreAll(
>>>> > > HBaseDStreamFunctionsSuite.scala:30)
>>>> > >   ...
>>>> > >   Cause: java.lang.RuntimeException: Failed construction of Master:
>>>> class
>>>> > > org.apache.hadoop.hbase.master.HMasterorg.apache.
>>>> > > hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > NativeStaticallyReferencedJniM
>>>> > > ethods.epollin()I
>>>> > >   at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(
>>>> > > JVMClusterUtil.java:145)
>>>> > >   at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(
>>>> > > LocalHBaseCluster.java:217)
>>>> > >   at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(
>>>> > > LocalHBaseCluster.java:152)
>>>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.init(
>>>> > > MiniHBaseCluster.java:214)
>>>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(
>>>> > > MiniHBaseCluster.java:94)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCl
>>>> uster(
>>>> > > HBaseTestingUtility.java:1124)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:1078)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:949)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:943)
>>>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>>>> > > HBaseTestingUtility.java:872)
>>>> > >   ...
>>>> > >   Cause: java.lang.UnsatisfiedLinkError: failed to load the required
>>>> > > native
>>>> > > library
>>>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > > Epoll.ensureAvailability(Epoll.java:78)
>>>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > > EpollEventLoopGroup.<clinit>(EpollEventLoopGroup.java:38)
>>>> > >   at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>(
>>>> > > NettyEventLoopGroupConfig.java:61)
>>>> > >   at org.apache.hadoop.hbase.regionserver.HRegionServer.<
>>>> > > init>(HRegionServer.java:552)
>>>> > >   at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:475)
>>>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> > Method)
>>>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>>>> > > NativeConstructorAccessorImpl.java:62)
>>>> > >   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>>>> > > DelegatingConstructorAccessorImpl.java:45)
>>>> > >   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>> > >   at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(
>>>> > > JVMClusterUtil.java:140)
>>>> > >   ...
>>>> > >   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.hbase.
>>>> > > shaded.io.netty.channel.epoll.NativeStaticallyReferencedJniM
>>>> > > ethods.epollin()I
>>>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > > NativeStaticallyReferencedJniMethods.epollin(Native Method)
>>>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > > Native.<clinit>(Native.java:66)
>>>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > > Epoll.<clinit>(Epoll.java:33)
>>>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>>>> > > EpollEventLoopGroup.<clinit>(EpollEventLoopGroup.java:38)
>>>> > >   at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>(
>>>> > > NettyEventLoopGroupConfig.java:61)
>>>> > >   at org.apache.hadoop.hbase.regionserver.HRegionServer.<
>>>> > > init>(HRegionServer.java:552)
>>>> > >   at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:475)
>>>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> > Method)
>>>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>>>> > > NativeConstructorAccessorImpl.java:62)
>>>> > >   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>>>> > > DelegatingConstructorAccessorImpl.java:45)
>>>> > >   ...
>>>> > > [INFO] ------------------------------------------------------------
>>>> > > ------------
>>>> > > [INFO] Reactor Summary:
>>>> > > [INFO]
>>>> > > [INFO] Apache HBase ....................................... SUCCESS
>>>> [
>>>> > > 1.575 s]
>>>> > > [INFO] Apache HBase - Checkstyle .......................... SUCCESS
[
>>>> > > 0.317 s]
>>>> > > [INFO] Apache HBase - Annotations ......................... SUCCESS
[
>>>> > > 0.537 s]
>>>> > > [INFO] Apache HBase - Build Configuration ................. SUCCESS
[
>>>> > > 0.053 s]
>>>> > > [INFO] Apache HBase - Shaded Protocol ..................... SUCCESS
[
>>>> > > 15.410 s]
>>>> > > [INFO] Apache HBase - Common .............................. SUCCESS
[
>>>> > > 4.603 s]
>>>> > > [INFO] Apache HBase - Metrics API ......................... SUCCESS
[
>>>> > > 1.213 s]
>>>> > > [INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS
[
>>>> > > 0.985 s]
>>>> > > [INFO] Apache HBase - Metrics Implementation .............. SUCCESS
[
>>>> > > 0.863 s]
>>>> > > [INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS
[
>>>> > > 1.750 s]
>>>> > > [INFO] Apache HBase - Protocol ............................ SUCCESS
[
>>>> > > 4.880 s]
>>>> > > [INFO] Apache HBase - Client .............................. SUCCESS
[
>>>> > > 5.233 s]
>>>> > > [INFO] Apache HBase - Replication ......................... SUCCESS
[
>>>> > > 1.040 s]
>>>> > > [INFO] Apache HBase - Prefix Tree ......................... SUCCESS
[
>>>> > > 1.121 s]
>>>> > > [INFO] Apache HBase - Procedure ........................... SUCCESS
[
>>>> > > 1.084 s]
>>>> > > [INFO] Apache HBase - Resource Bundle ..................... SUCCESS
[
>>>> > > 0.092 s]
>>>> > > [INFO] Apache HBase - Server .............................. SUCCESS
[
>>>> > > 19.849 s]
>>>> > > [INFO] Apache HBase - MapReduce ........................... SUCCESS
[
>>>> > > 4.221 s]
>>>> > > [INFO] Apache HBase - Testing Util ........................ SUCCESS
[
>>>> > > 3.273 s]
>>>> > > [INFO] Apache HBase - Thrift .............................. SUCCESS
[
>>>> > > 5.519 s]
>>>> > > [INFO] Apache HBase - RSGroup ............................. SUCCESS
[
>>>> > > 3.408 s]
>>>> > > [INFO] Apache HBase - Shell ............................... SUCCESS
>>>> [
>>>> > > 3.859 s]
>>>> > > [INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS
[
>>>> > > 4.038 s]
>>>> > > [INFO] Apache HBase - Backup .............................. SUCCESS
>>>> > [01:13
>>>> > > min]
>>>> > > [INFO] Apache HBase - Integration Tests ................... SUCCESS
[
>>>> > > 4.229 s]
>>>> > > [INFO] Apache HBase - Examples ............................ SUCCESS
[
>>>> > > 3.471 s]
>>>> > > [INFO] Apache HBase - Rest ................................ SUCCESS
>>>> [
>>>> > > 4.448 s]
>>>> > > [INFO] Apache HBase - External Block Cache ................ SUCCESS
[
>>>> > > 2.040 s]
>>>> > > [INFO] Apache HBase - Spark ............................... FAILURE
>>>> [
>>>> > > 32.833 s]
>>>> > > [INFO] Apache HBase - Spark Integration Tests ............. SKIPPED
>>>> > > [INFO] Apache HBase - Assembly ............................ SKIPPED
>>>> > > [INFO] Apache HBase - Shaded .............................. SKIPPED
>>>> > > [INFO] Apache HBase - Shaded - Client ..................... SKIPPED
>>>> > > [INFO] Apache HBase - Shaded - MapReduce .................. SKIPPED
>>>> > > [INFO] Apache HBase Shaded Packaging Invariants ........... SKIPPED
>>>> > > [INFO] Apache HBase - Archetypes .......................... SKIPPED
>>>> > > [INFO] Apache HBase - Exemplar for hbase-client archetype . SKIPPED
>>>> > > [INFO] Apache HBase - Exemplar for hbase-shaded-client archetype
>>>> SKIPPED
>>>> > > [INFO] Apache HBase - Archetype builder ................... SKIPPED
>>>> > > [INFO] ------------------------------------------------------------
>>>> > > ------------
>>>> > > [INFO] BUILD FAILURE
>>>> > > [INFO] ------------------------------------------------------------
>>>> > > ------------
>>>> > > [INFO] Total time: 03:26 min
>>>> > > [INFO] Finished at: 2017-09-27T19:34:35+05:30
>>>> > > [INFO] Final Memory: 345M/6055M
>>>> > > [INFO] ------------------------------------------------------------
>>>> > > ------------
>>>> > > [ERROR] Failed to execute goal org.scalatest:scalatest-maven-
>>>> > > plugin:1.0:test
>>>> > > (test) on project hbase-spark: There are test failures -> [Help
1]
>>>> > > [ERROR]
>>>> > > [ERROR] To see the full stack trace of the errors, re-run Maven
with
>>>> the
>>>> > -e
>>>> > > switch.
>>>> > > [ERROR] Re-run Maven using the -X switch to enable full debug
>>>> logging.
>>>> > > [ERROR]
>>>> > > [ERROR] For more information about the errors and possible solutions,
>>>> > > please read the following articles:
>>>> > > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/
>>>> > > MojoFailureException
>>>> > > [ERROR]
>>>> > > [ERROR] After correcting the problems, you can resume the build
with
>>>> the
>>>> > > command
>>>> > > [ERROR]   mvn <goals> -rf :hbase-spark
>>>> > >
>>>> >
>>>>
>>>
>>>
>>

Mime
View raw message