hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Compilation error: HBASE 0.98.4 with Snappy
Date Tue, 26 Aug 2014 15:02:08 GMT
Hi Arthur!

Our Snappy build instructions are currently out of date and I'm working on
updating them[1]. In short, I don't think there are any special build steps
for using snappy.

I'm still working out what needs to be included in our instructions for
local and cluster testing.

If you use the test for compression options, locally things will fail
because the native hadoop libs won't be present:

bin/hbase org.apache.hadoop.hbase.util.CompressionTest
file://*/tmp/snappy-test* snappy

(for comparison, replace "snappy" with "gz" and you will get a warning
about not having native libraries, but the test will succeed.)


I believe JM's suggestion is for you to copy the Hadoop native libraries
into the local HBase lib/native directory, which would allow the local test
to pass. If you are running in a deployed Hadoop cluster, I would expect
the necessary libraries to already be available to HBase.

[1]: https://issues.apache.org/jira/browse/HBASE-6189

-Sean


On Tue, Aug 26, 2014 at 8:30 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:

> Hi JM
>
> Below are my commands, tried two cases under same source code folder:
> a) compile with snappy parameters(failed),
> b) compile without snappy parameters (successful).
>
> Regards
> Arthur
>
> wget http://mirrors.devlib.org/apache/hbase/stable/hbase-0.98.4-src.tar.gz
> tar -vxf hbase-0.98.4-src.tar.gz
> mv hbase-0.98.4 hbase-0.98.4-src_snappy
> cd  hbase-0.98.4-src_snappy
> nano dev-support/generate-hadoopX-poms.sh
>   (change  hbase_home=“/usr/local/hadoop/hbase-0.98.4-src_snappy”)
>
>
> bash -x ./dev-support/generate-hadoopX-poms.sh 0.98.4 0.98.4-hadoop2
> a) with snappy parameters
> mvn -f pom.xml.hadoop2 install -DskipTests assembly:single
> -Prelease,hadoop-snappy -Dhadoop-snappy.version=0.0.1-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Building HBase - Server 0.98.4-hadoop2
> [INFO]
> ------------------------------------------------------------------------
> [WARNING] The POM for org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT
> is missing, no dependency information available
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] HBase ............................................. SUCCESS [8.192s]
> [INFO] HBase - Common .................................... SUCCESS [5.638s]
> [INFO] HBase - Protocol .................................. SUCCESS [1.535s]
> [INFO] HBase - Client .................................... SUCCESS [1.206s]
> [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.193s]
> [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [0.798s]
> [INFO] HBase - Prefix Tree ............................... SUCCESS [0.438s]
> [INFO] HBase - Server .................................... FAILURE [0.234s]
> [INFO] HBase - Testing Util .............................. SKIPPED
> [INFO] HBase - Thrift .................................... SKIPPED
> [INFO] HBase - Shell ..................................... SKIPPED
> [INFO] HBase - Integration Tests ......................... SKIPPED
> [INFO] HBase - Examples .................................. SKIPPED
> [INFO] HBase - Assembly .................................. SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 19.474s
> [INFO] Finished at: Tue Aug 26 21:21:13 HKT 2014
> [INFO] Final Memory: 51M/1100M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal on project hbase-server: Could not resolve
> dependencies for project org.apache.hbase:hbase-server:jar:0.98.4-hadoop2:
> Failure to find org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT in
> http://maven.oschina.net/content/groups/public/ was cached in the local
> repository, resolution will not be reattempted until the update interval of
> nexus-osc has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn <goals> -rf :hbase-server
>
>
>
>
> b) try again, without snappy parameters
> mvn -f pom.xml.hadoop2 install -DskipTests assembly:single -Prelease
> [INFO] Building tar:
> /edh/hadoop_all_sources/hbase-0.98.4-src_snappy/hbase-assembly/target/hbase-0.98.4-hadoop2-bin.tar.gz
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] HBase ............................................. SUCCESS [3.290s]
> [INFO] HBase - Common .................................... SUCCESS [3.119s]
> [INFO] HBase - Protocol .................................. SUCCESS [0.972s]
> [INFO] HBase - Client .................................... SUCCESS [0.920s]
> [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.167s]
> [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [0.504s]
> [INFO] HBase - Prefix Tree ............................... SUCCESS [0.382s]
> [INFO] HBase - Server .................................... SUCCESS [4.790s]
> [INFO] HBase - Testing Util .............................. SUCCESS [0.598s]
> [INFO] HBase - Thrift .................................... SUCCESS [1.536s]
> [INFO] HBase - Shell ..................................... SUCCESS [0.369s]
> [INFO] HBase - Integration Tests ......................... SUCCESS [0.443s]
> [INFO] HBase - Examples .................................. SUCCESS [0.459s]
> [INFO] HBase - Assembly .................................. SUCCESS
> [13.240s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 31.408s
> [INFO] Finished at: Tue Aug 26 21:22:50 HKT 2014
> [INFO] Final Memory: 57M/1627M
> [INFO]
> ------------------------------------------------------------------------
>
>
>
>
>
> On 26 Aug, 2014, at 8:52 pm, Jean-Marc Spaggiari <jean-marc@spaggiari.org>
> wrote:
>
> > Hi Arthur,
> >
> > How have you extracted HBase source and what command do you run to
> build? I
> > will do the same here locally so I can provide you the exact step to
> > complete.
> >
> > JM
> >
> >
> > 2014-08-26 8:42 GMT-04:00 Arthur.hk.chan@gmail.com <
> arthur.hk.chan@gmail.com
> >> :
> >
> >> Hi JM
> >>
> >> Not too sure what you mean, do you mean I should create a new folder in
> my
> >> HBASE_SRC named lib/native/Linux-x86 and copy these files to this folder
> >> then try to compile it again?
> >>
> >> Regards
> >> ARthur
> >>
> >>
> >> On 26 Aug, 2014, at 8:17 pm, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org>
> >> wrote:
> >>
> >>> Hi Arthur,
> >>>
> >>> Almost done! You now need to copy them on the HBase folder.
> >>>
> >>> hbase@hbasetest1:~/hbase-0.98.2-hadoop2/lib$ tree | grep -v .jar |
> grep
> >> -v
> >>> .rb
> >>> .
> >>> ├── native
> >>> │   └── Linux-x86
> >>> │       ├── libsnappy.a
> >>> │       ├── libsnappy.la
> >>> │       ├── libsnappy.so
> >>> │       ├── libsnappy.so.1
> >>> │       └── libsnappy.so.1.2.0
> >>>
> >>> I don't have any hadoop-snappy lib in my hbase folder and it works very
> >>> well with Snappy for me...
> >>>
> >>> JM
> >>>
> >>> 2014-08-26 8:09 GMT-04:00 Arthur.hk.chan@gmail.com <
> >> arthur.hk.chan@gmail.com
> >>>> :
> >>>
> >>>> Hi JM,
> >>>>
> >>>> Below are my steps to install snappy lib, do I miss something?
> >>>>
> >>>> Regards
> >>>> Arthur
> >>>>
> >>>> wget https://snappy.googlecode.com/files/snappy-1.1.1.tar.gz
> >>>> tar -vxf snappy-1.1.1.tar.gz
> >>>> cd snappy-1.1.1
> >>>> ./configure
> >>>> make
> >>>> make install
> >>>>       make[1]: Entering directory
> >> `/edh/hadoop_all_sources/snappy-1.1.1'
> >>>>       test -z "/usr/local/lib" || /bin/mkdir -p "/usr/local/lib"
> >>>>        /bin/sh ./libtool   --mode=install /usr/bin/install -c
> >>>> libsnappy.la '/usr/local/lib'
> >>>>       libtool: install: /usr/bin/install -c .libs/libsnappy.so.1.2.0
> >>>> /usr/local/lib/libsnappy.so.1.2.0
> >>>>       libtool: install: (cd /usr/local/lib && { ln -s -f
> >>>> libsnappy.so.1.2.0 libsnappy.so.1 || { rm -f libsnappy.so.1 &&
ln -s
> >>>> libsnappy.so.1.2.0 libsnappy.so.1; }; })
> >>>>       libtool: install: (cd /usr/local/lib && { ln -s -f
> >>>> libsnappy.so.1.2.0 libsnappy.so || { rm -f libsnappy.so && ln
-s
> >>>> libsnappy.so.1.2.0 libsnappy.so; }; })
> >>>>       libtool: install: /usr/bin/install -c .libs/libsnappy.lai
> >>>> /usr/local/lib/libsnappy.la
> >>>>       libtool: install: /usr/bin/install -c .libs/libsnappy.a
> >>>> /usr/local/lib/libsnappy.a
> >>>>       libtool: install: chmod 644 /usr/local/lib/libsnappy.a
> >>>>       libtool: install: ranlib /usr/local/lib/libsnappy.a
> >>>>       libtool: finish:
> >>>>
> >>
> PATH="/edh/hadoop/spark/bin:/edh/hadoop/hbase/bin:/edh/hadoop/zookeeper//bin:/edh/hadoop/yarn/hadoop/bin:/edh/hadoop/yarn/hadoop/sbin:/usr/lib64/qt-3.3/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/edh/hadoop/zookeeper//bin:/edh/hadoop/hive//bin:/usr/lib/jvm/jdk1.6.0_45//bin:/root/bin:/sbin"
> >>>> ldconfig -n /usr/local/lib
> >>>>
> >>>> ----------------------------------------------------------------------
> >>>>       Libraries have been installed in:
> >>>>       /usr/local/lib
> >>>>       If you ever happen to want to link against installed libraries
> >>>>       in a given directory, LIBDIR, you must either use libtool, and
> >>>>       specify the full pathname of the library, or use the `-LLIBDIR'
> >>>>       flag during linking and do at least one of the following:
> >>>>       - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
> >>>>       during execution
> >>>>       - add LIBDIR to the `LD_RUN_PATH' environment variable
> >>>>       during linking
> >>>>       - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
> >>>>       - have your system administrator add LIBDIR to `/etc/ld.so.conf'
> >>>>       See any operating system documentation about shared libraries
> for
> >>>>       more information, such as the ld(1) and ld.so(8) manual pages.
> >>>>
> >>>> ----------------------------------------------------------------------
> >>>>       test -z "/usr/local/share/doc/snappy" || /bin/mkdir -p
> >>>> "/usr/local/share/doc/snappy"
> >>>>        /usr/bin/install -c -m 644 ChangeLog COPYING INSTALL NEWS
> README
> >>>> format_description.txt framing_format.txt
> '/usr/local/share/doc/snappy'
> >>>>       test -z "/usr/local/include" || /bin/mkdir -p
> >> "/usr/local/include"
> >>>>        /usr/bin/install -c -m 644 snappy.h snappy-sinksource.h
> >>>> snappy-stubs-public.h snappy-c.h '/usr/local/include'
> >>>>       make[1]: Leaving directory
> `/edh/hadoop_all_sources/snappy-1.1.1'
> >>>>
> >>>> ll /usr/local/lib
> >>>>       -rw-r--r--. 1 root root   233554 Aug 20 00:14 libsnappy.a
> >>>>       -rwxr-xr-x. 1 root root      953 Aug 20 00:14 libsnappy.la
> >>>>       lrwxrwxrwx. 1 root root       18 Aug 20 00:14 libsnappy.so ->
> >>>> libsnappy.so.1.2.0
> >>>>       lrwxrwxrwx. 1 root root       18 Aug 20 00:14 libsnappy.so.1 ->
> >>>> libsnappy.so.1.2.0
> >>>>       -rwxr-xr-x. 1 root root   147726 Aug 20 00:14 libsnappy.so.1.2.0
> >>>>
> >>>>
> >>>>
> >>>> On 26 Aug, 2014, at 7:38 pm, Jean-Marc Spaggiari <
> >> jean-marc@spaggiari.org>
> >>>> wrote:
> >>>>
> >>>>> Hi Arthur,
> >>>>>
> >>>>> Do you have snappy libs installed and configured? HBase doesn't
come
> >> with
> >>>>> Snappy. So yo need to have it first.
> >>>>>
> >>>>> Shameless plug:
> >>>>>
> >>>>
> >>
> http://www.spaggiari.org/index.php/hbase/how-to-install-snappy-with-1#.U_xxSqdZuZY
> >>>>>
> >>>>> This is for 0.96 but should be very similar for 0.98. I will try
it
> >> soon
> >>>>> and post and update, but keep us posted here so we can support you...
> >>>>>
> >>>>> JM
> >>>>>
> >>>>>
> >>>>> 2014-08-26 7:34 GMT-04:00 Arthur.hk.chan@gmail.com <
> >>>> arthur.hk.chan@gmail.com
> >>>>>> :
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I need to install snappy to HBase 0.98.4.  (my Hadoop version
is
> >> 2.4.1)
> >>>>>>
> >>>>>> Can you please advise what would be wrong?  Should my pom.xml
be
> >>>> incorrect
> >>>>>> and missing something?
> >>>>>>
> >>>>>> Regards
> >>>>>> Arthur
> >>>>>>
> >>>>>>
> >>>>>> Below are my commands:
> >>>>>> bash -x ./dev-support/generate-hadoopX-poms.sh 0.98.4 0.98.4-hadoop2
> >>>>>> mvn -f pom.xml.hadoop2 install -DskipTests assembly:single
> >>>>>> -Prelease,hadoop-snappy
> >>>>>>
> >>>>>> Iog:
> >>>>>> [INFO]
> >>>>>>
> >> ------------------------------------------------------------------------
> >>>>>> [INFO] Building HBase - Server 0.98.4-hadoop2
> >>>>>> [INFO]
> >>>>>>
> >> ------------------------------------------------------------------------
> >>>>>> [WARNING] The POM for
> >> org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT
> >>>>>> is missing, no dependency information available
> >>>>>> [INFO]
> >>>>>>
> >> ------------------------------------------------------------------------
> >>>>>> [INFO] Reactor Summary:
> >>>>>> [INFO]
> >>>>>> [INFO] HBase ............................................. SUCCESS
> >>>> [3.129s]
> >>>>>> [INFO] HBase - Common .................................... SUCCESS
> >>>> [3.105s]
> >>>>>> [INFO] HBase - Protocol .................................. SUCCESS
> >>>> [0.976s]
> >>>>>> [INFO] HBase - Client .................................... SUCCESS
> >>>> [0.925s]
> >>>>>> [INFO] HBase - Hadoop Compatibility ...................... SUCCESS
> >>>> [0.183s]
> >>>>>> [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS
> >>>> [0.497s]
> >>>>>> [INFO] HBase - Prefix Tree ............................... SUCCESS
> >>>> [0.407s]
> >>>>>> [INFO] HBase - Server .................................... FAILURE
> >>>> [0.103s]
> >>>>>> [INFO] HBase - Testing Util .............................. SKIPPED
> >>>>>> [INFO] HBase - Thrift .................................... SKIPPED
> >>>>>> [INFO] HBase - Shell ..................................... SKIPPED
> >>>>>> [INFO] HBase - Integration Tests ......................... SKIPPED
> >>>>>> [INFO] HBase - Examples .................................. SKIPPED
> >>>>>> [INFO] HBase - Assembly .................................. SKIPPED
> >>>>>> [INFO]
> >>>>>>
> >> ------------------------------------------------------------------------
> >>>>>> [INFO] BUILD FAILURE
> >>>>>> [INFO]
> >>>>>>
> >> ------------------------------------------------------------------------
> >>>>>> [INFO] Total time: 9.939s
> >>>>>> [INFO] Finished at: Tue Aug 26 19:23:14 HKT 2014
> >>>>>> [INFO] Final Memory: 61M/2921M
> >>>>>> [INFO]
> >>>>>>
> >> ------------------------------------------------------------------------
> >>>>>> [ERROR] Failed to execute goal on project hbase-server: Could
not
> >>>> resolve
> >>>>>> dependencies for project
> >>>> org.apache.hbase:hbase-server:jar:0.98.4-hadoop2:
> >>>>>> Failure to find org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT
> in
> >>>>>> http://maven.oschina.net/content/groups/public/ was cached in
the
> >> local
> >>>>>> repository, resolution will not be reattempted until the update
> >>>> interval of
> >>>>>> nexus-osc has elapsed or updates are forced -> [Help 1]
> >>>>>> [ERROR]
> >>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven
with
> >> the
> >>>>>> -e switch.
> >>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> >>>>>> [ERROR]
> >>>>>> [ERROR] For more information about the errors and possible
> solutions,
> >>>>>> please read the following articles:
> >>>>>> [ERROR] [Help 1]
> >>>>>>
> >>>>
> >>
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> >>>>>> [ERROR]
> >>>>>> [ERROR] After correcting the problems, you can resume the build
with
> >> the
> >>>>>> command
> >>>>>> [ERROR]   mvn <goals> -rf :hbase-server
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>


-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message