hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prashant Kommireddi <prash1...@gmail.com>
Subject Re: how to integrate snappy into hadoop
Date Thu, 08 Dec 2011 02:12:54 GMT
I had to struggle a bit while building Snappy for Hadoop 0.20.2 on Ubuntu.
However, I have now been able to install it on a 10 node cluster and it
works great for map output compression. Please check these notes, may be it
might help in addition to official Hadoop-Snappy notes.

Tom White (amongst a few others) is the best person to answer this since he
is working directly with this integration project.

*Installing Maven 3*
http://www.discursive.com/blog/4636

*Snappy build*
http://code.google.com/p/hadoop-snappy/

*Creating the symbolic link for BUILD to pass:* THIS IS IMPORTANT
sudo ln -s
/home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/jre/lib/amd64/server/libjvm.so
/usr/local/lib/


*Additional build notes:*
http://shanky.org/2011/10/17/build-hadoop-from-source/

Install commands I issued on localhost, these were notes for myself. There
are dependencies that Snappy build needs, and you might NOT need all of the
commands that I have issues below. Please refer to official notes and
install accordingly.

pkommireddi@pkommireddi-wsl:~$ history | grep install
 1678  sudo apt-get install python-software-properties
 1681  sudo apt-get install maven
 1750  make install
 1751  ./configure && make && sudo make install
 1752  sudo apt-get install zlibc zlib1g zlib1g-dev
 1763  sudo apt-get install subversion
 1776  ./configure && make && sudo make install
 1824  sudo apt-get install zlibc zlib1g zlib1g-dev
 1846  make install
 1847  sudo make install
 1858  make installcheck
 1861  sudo make install
 1862  sudo make installcheck
 1904  sudo apt-get install libtool
 1907  sudo apt-get install automake

Thanks,
Prashant Kommireddi

On Wed, Dec 7, 2011 at 5:39 PM, Jinyan Xu <Jinyan.Xu@exar.com> wrote:

> Hi ,
>
>
> Anyone else have the experience integrating snappy into hadoop ?  help me
> with it....
>
> I find google doesn't provide the hadoop-snappy now :
>
> "Hadoop-snappy is integrated into Hadoop Common(JUN 2011).
>
> Hadoop-Snappy can be used as an add-on for recent (released) versions of
> Hadoop that do not provide Snappy Codec support yet.
>
> Hadoop-Snappy is being kept in synch with Hadoop Common. "
>
>
>
>
>
> Thanks!
>
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message