bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 김영우 <warwit...@gmail.com>
Subject Re: Bundling Snappy with Hadoop
Date Thu, 11 Jun 2015 05:21:06 GMT
Snappy itself is one of the widely used compression codec for Hadoop and
it's ecosystem. Hadoop distros like CDH and HDP have snappy support by
default and provide related docs.

IMO, If Hadoop in Bigtop has support snappy by default, we should include
it in bigtop_toolchain *explicitly* to avoid confusion.

Thanks,
Youngwoo

On Thu, Jun 11, 2015 at 5:28 AM, Olaf Flebbe <of@oflebbe.de> wrote:

> Hi Evans,
>
> I see no essential need for snappy, too. If we do the CI build with
> snappy, than we rather should include it in bigtop_toolchain.
>
> Maybe it would be better to include snappy in an option puppet manifest
> rather to hardcode it in packages.pp. Feel free to reject my patch in the
> JIRA optionally with proposing a different solution.
>
> Olaf
>
>
>
> > Am 10.06.2015 um 03:10 schrieb Evans Ye <evansye@apache.org>:
> >
> > Thanks for the comment Olaf.
> > Whether or not to include snappy codec is optional in upstream hadoop
> projrct, therefore I'm not quite sure to put it in to toolchain is the
> right way to go. Since this implies that we'll build hadoop with snappy by
> default.
> > Do you see most of your users or customer require this feature?
> >
> > 2015/6/10 上午4:13 於 "Olaf Flebbe" <of@oflebbe.de> 寫道:
> > Hi Kelly,
> >
> > the bigtop-toolchain needs more love for amazon-linux. There is already
> a JIRA for it : BIGTOP-1883 , but it should be improved slightly.
> >
> > Olaf
> >
> >
> >
> > > Am 09.06.2015 um 21:35 schrieb Kelly, Jonathan <jonathak@amazon.com>:
> > >
> > > Oh, OK, do you guys want it in the toolchain for all distros by
> default? I
> > > wasn't sure if you did (I probably should have asked), so for now I
> only
> > > added it to AmazonLinux in our own repo. If you'd like to add it to all
> > > distros, that would be great.
> > >
> > > Thanks,
> > > Jonathan Kelly
> > >
> > >
> > >
> > >
> > > On 6/9/15, 11:56 AM, "Olaf Flebbe" <of@oflebbe.de> wrote:
> > >
> > >> Hi Evans,
> > >>
> > >> Just discovered that snappy-devel is missing from bigtop_toolchain.
> Will
> > >> open a JIRA for it.
> > >>
> > >> Olaf
> > >>
> > >>
> > >>> Am 09.06.2015 um 05:13 schrieb Evans Ye <evansye@apache.org>:
> > >>>
> > >>> Yup, you need snappy-devel to be installed. Our new distribution
> > >>> targeting bigtop 1.0 is building hadoop with snappy, so the snappy
> > >>> compression should work at your side as well. :)
> > >>>
> > >>> 2015-06-09 6:57 GMT+08:00 Kelly, Jonathan <jonathak@amazon.com>:
> > >>> Nvm, answered my own question. Apparently I want snappy-devel rather
> > >>> than snappy.
> > >>>
> > >>> ~ Jonathan Kelly
> > >>>
> > >>> From: <Kelly>, Jonathan Kelly <jonathak@amazon.com>
> > >>> Date: Monday, June 8, 2015 at 3:31 PM
> > >>> To: "user@bigtop.apache.org" <user@bigtop.apache.org>
> > >>> Subject: Bundling Snappy with Hadoop
> > >>>
> > >>> Is anybody bundling Hadoop with Snappy support on Bigtop? I noticed
> > >>> that
> > >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/h
> > >>> adoop/do-component-build checks for the presence of
> > >>> /usr/lib/libsnappy.so or /usr/lib64/libsnappy.so and sets
> $BUNDLE_SNAPPY
> > >>> accordingly, so it seems that I should just need to install the
> snappy
> > >>> package, and Hadoop will automatically be built with Snappy support.
> > >>> However, when I install the snappy package ("sudo yum install -y
> snappy"
> > >>> on AmazonLinux 2015.03), the library file that gets installed is
> > >>> /usr/lib64/libsnappy.so, not /usr/lib64/libsnappy.so.1, so installing
> > >>> the snappy package has no effect on the Hadoop build.
> > >>>
> > >>> Thanks,
> > >>> Jonathan Kelly
> > >>>
> > >>
> > >
> >
>
>

Mime
View raw message