hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinyan Xu <Jinyan...@exar.com>
Subject RE: how to integrate snappy into hadoop
Date Thu, 08 Dec 2011 03:18:16 GMT
Hi Prashant Kommireddi,

Last week, I read build-hadoop-from-source and follow it, but I failed to compile hbase with
"mvn compile -Dsnappy ". Did you install HBase 0.90.2?
According build-hadoop-from-source HBase 0.90.2 is incompatible with Hadoop0.20.2-release.


-----Original Message-----
From: Prashant Kommireddi [mailto:prash1784@gmail.com]
Sent: 2011年12月8日 10:13
To: common-user@hadoop.apache.org
Subject: Re: how to integrate snappy into hadoop

I had to struggle a bit while building Snappy for Hadoop 0.20.2 on Ubuntu.
However, I have now been able to install it on a 10 node cluster and it
works great for map output compression. Please check these notes, may be it
might help in addition to official Hadoop-Snappy notes.

Tom White (amongst a few others) is the best person to answer this since he
is working directly with this integration project.

*Installing Maven 3*
http://www.discursive.com/blog/4636

*Snappy build*
http://code.google.com/p/hadoop-snappy/

*Creating the symbolic link for BUILD to pass:* THIS IS IMPORTANT
sudo ln -s
/home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/jre/lib/amd64/server/libjvm.so
/usr/local/lib/


*Additional build notes:*
http://shanky.org/2011/10/17/build-hadoop-from-source/

Install commands I issued on localhost, these were notes for myself. There
are dependencies that Snappy build needs, and you might NOT need all of the
commands that I have issues below. Please refer to official notes and
install accordingly.

pkommireddi@pkommireddi-wsl:~$ history | grep install
 1678  sudo apt-get install python-software-properties
 1681  sudo apt-get install maven
 1750  make install
 1751  ./configure && make && sudo make install
 1752  sudo apt-get install zlibc zlib1g zlib1g-dev
 1763  sudo apt-get install subversion
 1776  ./configure && make && sudo make install
 1824  sudo apt-get install zlibc zlib1g zlib1g-dev
 1846  make install
 1847  sudo make install
 1858  make installcheck
 1861  sudo make install
 1862  sudo make installcheck
 1904  sudo apt-get install libtool
 1907  sudo apt-get install automake

Thanks,
Prashant Kommireddi

On Wed, Dec 7, 2011 at 5:39 PM, Jinyan Xu <Jinyan.Xu@exar.com> wrote:

> Hi ,
>
>
> Anyone else have the experience integrating snappy into hadoop ?  help me
> with it....
>
> I find google doesn't provide the hadoop-snappy now :
>
> "Hadoop-snappy is integrated into Hadoop Common(JUN 2011).
>
> Hadoop-Snappy can be used as an add-on for recent (released) versions of
> Hadoop that do not provide Snappy Codec support yet.
>
> Hadoop-Snappy is being kept in synch with Hadoop Common. "
>
>
>
>
>
> Thanks!
>
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>

The information and any attached documents contained in this message
may be confidential and/or legally privileged.  The message is
intended solely for the addressee(s).  If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful.  If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.
Mime
View raw message