hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: zstd compression
Date Tue, 25 Jul 2017 17:27:23 GMT
Thanks Sean, Owen. I've opened an issue on their github here:

https://github.com/facebook/zstd/issues/775

I figure it doesn't hurt to ask, particularly if they intend for zstd to be
a replacement for the commonly-embedded zlib.

On Tue, Jul 25, 2017 at 6:17 AM, Owen O'Malley <owen.omalley@gmail.com>
wrote:

> I'd support asking Facebook to change it with both my hadoop and orc hats
> on.
>
> .. Owen
>
> > On Jul 24, 2017, at 23:43, Sean Busbey <busbey@cloudera.com> wrote:
> >
> > Nope. Once I found out HBase's use was compliant as an optional runtime
> > dependency I stopped looking.
> >
> >> On Jul 24, 2017 7:22 PM, "Andrew Wang" <andrew.wang@cloudera.com>
> wrote:
> >>
> >> I think it'd still be worth asking FB to relicense zstandard. Being able
> >> to bundle it in the release would make it easier to use, since I doubt
> >> there are zstandard packages in the default OS repos.
> >>
> >> Sean, have you already filed an issue with zstandard?
> >>
> >> On Mon, Jul 17, 2017 at 1:30 PM, Jason Lowe <jlowe@yahoo-inc.com.invalid
> >
> >> wrote:
> >>
> >>> I think we are OK to leave support for the zstd codec in the Hadoop
> code
> >>> base.  I asked Chris Mattman for clarification, noting that the
> support for
> >>> the zstd codec requires the user to install the zstd headers and
> libraries
> >>> and then configure it to be included in the native Hadoop build.  The
> >>> Hadoop releases are not shipping any zstd code (e.g.: headers or
> libraries)
> >>> nor does it require zstd as a mandatory dependency.  Here's what he
> said:
> >>>
> >>>
> >>> On Monday, July 17, 2017 11:07 AM, Chris Mattmann <mattmann@apache.org
> >
> >>> wrote:
> >>>
> >>>> Hi Jason,
> >>>>
> >>>> This sounds like an optional dependency on a Cat-X software. This
> isn’t
> >>> the only type of compression
> >>>> that is allowed within Hadoop, correct? If it is truly optional and
> you
> >>> have gone to that level of detail
> >>>> below to make the user opt in, and if we are not shipping zstd with
> our
> >>> products (source code releases),
> >>>> then this is an acceptable usage.
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>
> >>>
> >>> So I think we are in the clear with respect to zstd usage as long as we
> >>> keep it as an optional codec where the user needs to get the headers
> and
> >>> libraries for zstd and configure it into the native Hadoop build.
> >>>
> >>> Jason
> >>>
> >>> On Monday, July 17, 2017 9:44 AM, Sean Busbey <busbey@cloudera.com>
> >>> wrote:
> >>>
> >>>
> >>>
> >>> I know that the HBase community is also looking at what to do about
> >>>
> >>> our inclusion of zstd. We've had it in releases since late 2016. My
> >>>
> >>> plan was to request that they relicense it.
> >>>
> >>>
> >>> Perhaps the Hadoop PMC could join HBase in the request?
> >>>
> >>>
> >>> On Sun, Jul 16, 2017 at 8:11 PM, Allen Wittenauer
> >>>
> >>> <aw@effectivemachines.com> wrote:
> >>>
> >>>>
> >>>
> >>>>        It looks like HADOOP-13578 added Facebook's zstd compression
> >>> codec.  Unfortunately, that codec is using the same 3-clause BSD
> (LICENSE
> >>> file) + patent grant license (PATENTS file) that React is using and
> RocksDB
> >>> was using.
> >>>
> >>>>
> >>>
> >>>>        Should that code get reverted?
> >>>
> >>>>
> >>>
> >>>>
> >>>
> >>>>
> >>>
> >>>> ---------------------------------------------------------------------
> >>>
> >>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>
> >>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> busbey
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>>
> >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>
> >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>>
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message