hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Error : java.lang.UnsatisfiedLinkError: failed to load the required native library for netty
Date Tue, 03 Oct 2017 17:03:19 GMT
The below gets us further but now I see that the spark tests take a really
long time to run on linux but complete promptly on macosx (2m 55s).
Looking....

St.Ack

On Tue, Oct 3, 2017 at 9:13 AM, Stack <stack@duboce.net> wrote:

> This seems to work for me. Does it work for you?
>
>
> diff --git a/hbase-spark/pom.xml b/hbase-spark/pom.xml
> index 594aa2a..6d191e3 100644
> --- a/hbase-spark/pom.xml
> +++ b/hbase-spark/pom.xml
> @@ -568,6 +568,9 @@
>            <junitxml>.</junitxml>
>            <filereports>WDF TestSuite.txt</filereports>
>            <parallel>false</parallel>
> +          <systemProperties>
> +            <org.apache.hadoop.hbase.shaded.io.netty.packagePrefix>
> org.apache.hadoop.hbase.shaded.</org.apache.hadoop.hbase.shaded.io.netty.
> packagePrefix>
> +          </systemProperties>
>          </configuration>
>          <executions>
>            <execution>
>
> St.Ack
>
> On Tue, Oct 3, 2017 at 8:45 AM, Amit Kabra <amitkabraiiit@gmail.com>
> wrote:
>
>> Thanks Stack / Sean Busbey for replying.
>>
>> OS : Ubuntu 16.04.2 , 64 bit.
>> Eclipse : Version: Neon.3 Release (4.6.3)
>> HBase branch : branch-2
>> Command line test to reproduce : mvn clean package
>> -Dtest=TestIncrementalBackup
>> Reproduce from eclipse , right click on TestIncBackupRestore and run as
>> junit from test class TestIncrementalBackup.
>> No I am not embedding hbase in my application. I have just checked out
>> hbase , switched to branch-2 and run the unit test from command line or
>> from eclipse. Failing with same error in both cases.
>> Yes the trailing period is also present.
>>
>> Thanks,
>> Amit Kabra.
>>
>>
>>
>>
>>
>> On Tue, Oct 3, 2017 at 8:53 PM, Stack <stack@duboce.net> wrote:
>>
>> > Thank you for the detail.
>> >
>> > Pardon the questions below asking for yet more detail. I am unable to
>> > reproduce locally or on another os (though we see this issue up on our
>> > build box).
>> >
>> > What is your OS when you see the below?
>> >
>> > On Tue, Oct 3, 2017 at 2:06 AM, Amit Kabra <amitkabraiiit@gmail.com>
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > I am using "branch-2" branch of hbase, when I run unit test I get
>> > following
>> > > error for netty "java.lang.UnsatisfiedLinkError: failed to load the
>> > > required native library"
>> > >
>> > >
>> > > This is running a unit test in your eclipse environment?
>> >
>> > You are trying to run an hbase-spark unit test when you see the above?
>> >
>> >
>> >
>> >
>> > > *I already have following set in "maven-surefire-plugin" in pom.xml as
>> > > per http://hbase.apache.org/book.html#thirdparty
>> > > <http://hbase.apache.org/book.html#thirdparty>*
>> > >
>> > >
>> > >
>> >
>> > Are you embedding hbase into your application?
>> >
>> >
>> >
>> > >             <systemPropertyVariables>
>> > >                 <!--
>> > >               <test.build.classes>${test.build.classes}</test.build.
>> > classe
>> > > s>
>> > >                 -->
>> > >               <!--For shaded netty, to find the relocated .so.
>> > >                    Trick from
>> > >                 https://stackoverflow.com/questions/33825743/rename-
>> > > files-inside-a-jar-using-some-maven-plugin
>> > > <https://stackoverflow.com/questions/33825743/rename-
>> > files-inside-a-jar-using-some-maven-plugin>
>> > >
>> > >                 The netty jar has a .so in it. Shading requires
>> rename of
>> > > the .so and then passing a system
>> > >                 property so netty finds the renamed .so and
>> associates it
>> > > w/ the relocated netty files.
>> > >
>> > >                 The relocated netty is in hbase-thirdparty dependency.
>> > Just
>> > > set this propery globally rather
>> > >                 than per module.
>> > >                -->
>> > >               <org.apache.hadoop.hbase.shaded.io.netty.packagePrefix>
>> > > org.apache.hadoop.hbase.shaded.</org.apache.hadoop.
>> > hbase.shaded.io.netty.
>> > > packagePrefix>
>> > >             </systemPropertyVariables>
>> > >
>> > >
>> > >
>> > > *And I see in the code as per HBASE-18271, all io.netty is already
>> > replaced
>> > > with org.apache.hadoop.hbase.shaded.io.netty*
>> > >
>> > >
>> > The trailing period is also present?
>> >
>> >
>> >
>> > >
>> > > If I run a test from eclipse , I see the error immediately and my test
>> > > doesn't run, but when I run from command line , the test runs but I
>> get
>> > the
>> > > error at the end when the mvn command finishes.
>> > >
>> > >
>> > > Is it any eclipse test?
>> >
>> > Thank you. Let me try and fix this this morning.
>> >
>> > S
>> >
>> >
>> >
>> >
>> >
>> > > *Here is the complete error output.*
>> > >
>> > >
>> > >
>> > > [INFO]
>> > > [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @
>> > hbase-spark
>> > > ---
>> > > [INFO]
>> > > [INFO] --- scalatest-maven-plugin:1.0:test (test) @ hbase-spark ---
>> > > Discovery starting.
>> > > Discovery completed in 1 second, 558 milliseconds.
>> > > Run starting. Expected test count is: 79
>> > > HBaseDStreamFunctionsSuite:
>> > > Formatting using clusterid: testClusterID
>> > > *** RUN ABORTED ***
>> > >   java.io.IOException: Shutting down
>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.init(
>> > > MiniHBaseCluster.java:232)
>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(
>> > > MiniHBaseCluster.java:94)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCl
>> uster(
>> > > HBaseTestingUtility.java:1124)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:1078)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:949)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:943)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:872)
>> > >   at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.bef
>> oreAll(
>> > > HBaseDStreamFunctionsSuite.scala:41)
>> > >   at org.scalatest.BeforeAndAfterAll$class.beforeAll(
>> BeforeAndAfterAll.
>> > > scala:187)
>> > >   at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.bef
>> oreAll(
>> > > HBaseDStreamFunctionsSuite.scala:30)
>> > >   ...
>> > >   Cause: java.lang.RuntimeException: Failed construction of Master:
>> class
>> > > org.apache.hadoop.hbase.master.HMasterorg.apache.
>> > > hadoop.hbase.shaded.io.netty.channel.epoll.
>> > NativeStaticallyReferencedJniM
>> > > ethods.epollin()I
>> > >   at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(
>> > > JVMClusterUtil.java:145)
>> > >   at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(
>> > > LocalHBaseCluster.java:217)
>> > >   at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(
>> > > LocalHBaseCluster.java:152)
>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.init(
>> > > MiniHBaseCluster.java:214)
>> > >   at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(
>> > > MiniHBaseCluster.java:94)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCl
>> uster(
>> > > HBaseTestingUtility.java:1124)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:1078)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:949)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:943)
>> > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
>> > > HBaseTestingUtility.java:872)
>> > >   ...
>> > >   Cause: java.lang.UnsatisfiedLinkError: failed to load the required
>> > > native
>> > > library
>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>> > > Epoll.ensureAvailability(Epoll.java:78)
>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>> > > EpollEventLoopGroup.<clinit>(EpollEventLoopGroup.java:38)
>> > >   at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>(
>> > > NettyEventLoopGroupConfig.java:61)
>> > >   at org.apache.hadoop.hbase.regionserver.HRegionServer.<
>> > > init>(HRegionServer.java:552)
>> > >   at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:475)
>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method)
>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>> > > NativeConstructorAccessorImpl.java:62)
>> > >   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>> > > DelegatingConstructorAccessorImpl.java:45)
>> > >   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> > >   at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(
>> > > JVMClusterUtil.java:140)
>> > >   ...
>> > >   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.hbase.
>> > > shaded.io.netty.channel.epoll.NativeStaticallyReferencedJniM
>> > > ethods.epollin()I
>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>> > > NativeStaticallyReferencedJniMethods.epollin(Native Method)
>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>> > > Native.<clinit>(Native.java:66)
>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>> > > Epoll.<clinit>(Epoll.java:33)
>> > >   at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
>> > > EpollEventLoopGroup.<clinit>(EpollEventLoopGroup.java:38)
>> > >   at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>(
>> > > NettyEventLoopGroupConfig.java:61)
>> > >   at org.apache.hadoop.hbase.regionserver.HRegionServer.<
>> > > init>(HRegionServer.java:552)
>> > >   at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:475)
>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method)
>> > >   at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>> > > NativeConstructorAccessorImpl.java:62)
>> > >   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>> > > DelegatingConstructorAccessorImpl.java:45)
>> > >   ...
>> > > [INFO] ------------------------------------------------------------
>> > > ------------
>> > > [INFO] Reactor Summary:
>> > > [INFO]
>> > > [INFO] Apache HBase ....................................... SUCCESS [
>> > > 1.575 s]
>> > > [INFO] Apache HBase - Checkstyle .......................... SUCCESS [
>> > > 0.317 s]
>> > > [INFO] Apache HBase - Annotations ......................... SUCCESS [
>> > > 0.537 s]
>> > > [INFO] Apache HBase - Build Configuration ................. SUCCESS [
>> > > 0.053 s]
>> > > [INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [
>> > > 15.410 s]
>> > > [INFO] Apache HBase - Common .............................. SUCCESS [
>> > > 4.603 s]
>> > > [INFO] Apache HBase - Metrics API ......................... SUCCESS [
>> > > 1.213 s]
>> > > [INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [
>> > > 0.985 s]
>> > > [INFO] Apache HBase - Metrics Implementation .............. SUCCESS [
>> > > 0.863 s]
>> > > [INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [
>> > > 1.750 s]
>> > > [INFO] Apache HBase - Protocol ............................ SUCCESS [
>> > > 4.880 s]
>> > > [INFO] Apache HBase - Client .............................. SUCCESS [
>> > > 5.233 s]
>> > > [INFO] Apache HBase - Replication ......................... SUCCESS [
>> > > 1.040 s]
>> > > [INFO] Apache HBase - Prefix Tree ......................... SUCCESS [
>> > > 1.121 s]
>> > > [INFO] Apache HBase - Procedure ........................... SUCCESS [
>> > > 1.084 s]
>> > > [INFO] Apache HBase - Resource Bundle ..................... SUCCESS [
>> > > 0.092 s]
>> > > [INFO] Apache HBase - Server .............................. SUCCESS [
>> > > 19.849 s]
>> > > [INFO] Apache HBase - MapReduce ........................... SUCCESS [
>> > > 4.221 s]
>> > > [INFO] Apache HBase - Testing Util ........................ SUCCESS [
>> > > 3.273 s]
>> > > [INFO] Apache HBase - Thrift .............................. SUCCESS [
>> > > 5.519 s]
>> > > [INFO] Apache HBase - RSGroup ............................. SUCCESS [
>> > > 3.408 s]
>> > > [INFO] Apache HBase - Shell ............................... SUCCESS [
>> > > 3.859 s]
>> > > [INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [
>> > > 4.038 s]
>> > > [INFO] Apache HBase - Backup .............................. SUCCESS
>> > [01:13
>> > > min]
>> > > [INFO] Apache HBase - Integration Tests ................... SUCCESS [
>> > > 4.229 s]
>> > > [INFO] Apache HBase - Examples ............................ SUCCESS [
>> > > 3.471 s]
>> > > [INFO] Apache HBase - Rest ................................ SUCCESS [
>> > > 4.448 s]
>> > > [INFO] Apache HBase - External Block Cache ................ SUCCESS [
>> > > 2.040 s]
>> > > [INFO] Apache HBase - Spark ............................... FAILURE [
>> > > 32.833 s]
>> > > [INFO] Apache HBase - Spark Integration Tests ............. SKIPPED
>> > > [INFO] Apache HBase - Assembly ............................ SKIPPED
>> > > [INFO] Apache HBase - Shaded .............................. SKIPPED
>> > > [INFO] Apache HBase - Shaded - Client ..................... SKIPPED
>> > > [INFO] Apache HBase - Shaded - MapReduce .................. SKIPPED
>> > > [INFO] Apache HBase Shaded Packaging Invariants ........... SKIPPED
>> > > [INFO] Apache HBase - Archetypes .......................... SKIPPED
>> > > [INFO] Apache HBase - Exemplar for hbase-client archetype . SKIPPED
>> > > [INFO] Apache HBase - Exemplar for hbase-shaded-client archetype
>> SKIPPED
>> > > [INFO] Apache HBase - Archetype builder ................... SKIPPED
>> > > [INFO] ------------------------------------------------------------
>> > > ------------
>> > > [INFO] BUILD FAILURE
>> > > [INFO] ------------------------------------------------------------
>> > > ------------
>> > > [INFO] Total time: 03:26 min
>> > > [INFO] Finished at: 2017-09-27T19:34:35+05:30
>> > > [INFO] Final Memory: 345M/6055M
>> > > [INFO] ------------------------------------------------------------
>> > > ------------
>> > > [ERROR] Failed to execute goal org.scalatest:scalatest-maven-
>> > > plugin:1.0:test
>> > > (test) on project hbase-spark: There are test failures -> [Help 1]
>> > > [ERROR]
>> > > [ERROR] To see the full stack trace of the errors, re-run Maven with
>> the
>> > -e
>> > > switch.
>> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> > > [ERROR]
>> > > [ERROR] For more information about the errors and possible solutions,
>> > > please read the following articles:
>> > > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/
>> > > MojoFailureException
>> > > [ERROR]
>> > > [ERROR] After correcting the problems, you can resume the build with
>> the
>> > > command
>> > > [ERROR]   mvn <goals> -rf :hbase-spark
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message