hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Hadoop encryption module as Apache Chimera incubator project
Date Thu, 04 Feb 2016 00:50:01 GMT
Let's do one step at a time. There is a clear need for common encryption,
and let's focus on making that happen.

On Wed, Feb 3, 2016 at 4:48 PM, Zheng, Kai <kai.zheng@intel.com> wrote:

> I thought this discussion would switch to common-dev@ now?
>
> >> Would it make sense to also package some of the compression libraries,
> and maybe some of the text processing from MapReduce? Evolving some of this
> code to a common library with few/no dependencies would be generally
> useful. As a subproject, it could have a broader scope that could evolve
> into a viable TLP.
>
> Sounds like a great idea to make the potential TLP more sense!! I thought
> it could be organized like in Apache common, the security, compression and
> other common text related things could be organized in different
> independent modules. Perhaps Hadoop conf could also be considered. These
> modules could rely on some common utility module. It can still be Hadoop
> background or powered, and eventually we would have a good place for some
> Hadoop common codes to move into to benefit and impact even more broad
> scope than Hadoop itself.
>
> Regards,
> Kai
>
> -----Original Message-----
> From: Chris Douglas [mailto:cdouglas@apache.org]
> Sent: Thursday, February 04, 2016 7:26 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: Hadoop encryption module as Apache Chimera incubator project
>
> I went through the repository, and now understand the reasoning that would
> locate this code in Apache Commons. This isn't proposing to extract much of
> the implementation and it takes none of the integration. It's limited to
> interfaces to crypto libraries and streams/configuration. It might be a
> reasonable fit for commons-codec, but that's a pretty sparse library and
> driving the release cadence might be more complicated. It'd be worth
> discussing on their lists (please also CC common-dev@).
>
> Chimera would be a boutique TLP, unless we wanted to draw out more of the
> integration and tooling. Is that a goal you're interested in pursuing?
> There's a tension between keeping this focused and including enough
> functionality to make it viable as an independent component. By way of
> example, Hadoop's common project requires too many dependencies and carries
> too much historical baggage for other projects to rely on.
> I agree with Colin/Steve: we don't want this to grow into another
> guava-like dependency that creates more work in conflicts than it saves in
> implementation...
>
> Would it make sense to also package some of the compression libraries, and
> maybe some of the text processing from MapReduce? Evolving some of this
> code to a common library with few/no dependencies would be generally
> useful. As a subproject, it could have a broader scope that could evolve
> into a viable TLP. If the encryption libraries are the only ones you're
> interested in pulling out, then Apache Commons does seem like a better
> target than a separate project. -C
>
>
> On Wed, Feb 3, 2016 at 1:49 AM, Chris Douglas <cdouglas@apache.org> wrote:
> > On Wed, Feb 3, 2016 at 12:48 AM, Gangumalla, Uma
> > <uma.gangumalla@intel.com> wrote:
> >>>Standing in the point of shared fundamental piece of code like this,
> >>>I do think Apache Commons might be the best direction which we can
> >>>try as the first effort. In this direction, we still need to work
> >>>with Apache Common community for buying in and accepting the proposal.
> >> Make sense.
> >
> > Makes sense how?
> >
> >> For this we should define the independent release cycles for this
> >> project and it would just place under Hadoop tree if we all conclude
> >> with this option at the end.
> >
> > Yes.
> >
> >> [Chris]
> >>>If Chimera is not successful as an independent project or stalls,
> >>>Hadoop and/or Spark and/or $project will have to reabsorb it as
> >>>maintainers.
> >>>
> >> I am not so strong on this point. If we assume project would be
> >> unsuccessful, it can be unsuccessful(less maintained) even under hadoop.
> >> But if other projects depending on this piece then they would get
> >> less support. Of course right now we feel this piece of code is very
> >> important and we feel(expect) it can be successful as independent
> >> project, irrespective of whether it as separate project outside hadoop
> or inside.
> >> So, I feel this point would not really influence to judge the
> discussion.
> >
> > Sure; code can idle anywhere, but that wasn't the point I was after.
> > You propose to extract code from Hadoop, but if Chimera fails then
> > what recourse do we have among the other projects taking a dependency
> > on it? Splitting off another project is feasible, but Chimera should
> > be sustainable before this PMC can divest itself of responsibility for
> > security libraries. That's a pretty low bar.
> >
> > Bundling the library with the jar is helpful; I've used that before.
> > It should prefer (updated) libraries from the environment, if
> > configured. Otherwise it's a pain (or impossible) for ops to patch
> > security bugs. -C
> >
> >>>-----Original Message-----
> >>>From: Colin P. McCabe [mailto:cmccabe@apache.org]
> >>>Sent: Wednesday, February 3, 2016 4:56 AM
> >>>To: hdfs-dev@hadoop.apache.org
> >>>Subject: Re: Hadoop encryption module as Apache Chimera incubator
> >>>project
> >>>
> >>>It's great to see interest in improving this functionality.  I think
> >>>Chimera could be successful as an Apache project.  I don't have a
> >>>strong opinion one way or the other as to whether it belongs as part
> >>>of Hadoop or separate.
> >>>
> >>>I do think there will be some challenges splitting this functionality
> >>>out into a separate jar, because of the way our CLASSPATH works right
> now.
> >>>For example, let's say that Hadoop depends on Chimera 1.2 and Spark
> >>>depends on Chimera 1.1.  Now Spark jobs have two different versions
> >>>fighting it out on the classpath, similar to the situation with Guava
> >>>and other libraries.  Perhaps if Chimera adopts a policy of strong
> >>>backwards compatibility, we can just always use the latest jar, but
> >>>it still seems likely that there will be problems.  There are various
> >>>classpath isolation ideas that could help here, but they are big
> >>>projects in their own right and we don't have a clear timeline for
> >>>them.  If this does end up being a separate jar, we may need to shade
> >>>it to avoid all these issues.
> >>>
> >>>Bundling the JNI glue code in the jar itself is an interesting idea,
> >>>which we have talked about before for libhadoop.so.  It doesn't
> >>>really have anything to do with the question of TLP vs. non-TLP, of
> course.
> >>>We could do that refactoring in Hadoop itself.  The really
> >>>complicated part of bundling JNI code in a jar is that you need to
> >>>create jars for every cross product of (JVM version, openssl version,
> operating system).
> >>>For example, you have the RHEL6 build for openJDK7 using openssl 1.0.1e.
> >>>If you change any one thing-- say, change openJDK7 to Oracle JDK8,
> >>>then you might need to rebuild.  And certainly using Ubuntu would be
> >>>a rebuild.  And so forth.  This kind of clashes with Maven's
> >>>philosophy of pulling prebuilt jars from the internet.
> >>>
> >>>Kai Zheng's question about whether we would bundle openSSL's
> >>>libraries is a good one.  Given the high rate of new vulnerabilities
> >>>discovered in that library, it seems like bundling would require
> >>>Hadoop users and vendors to update very frequently, much more
> >>>frequently than Hadoop is traditionally updated.  So probably we would
> not choose to bundle openssl.
> >>>
> >>>best,
> >>>Colin
> >>>
> >>>On Tue, Feb 2, 2016 at 12:29 AM, Chris Douglas <cdouglas@apache.org>
> >>>wrote:
> >>>> As a subproject of Hadoop, Chimera could maintain its own cadence.
> >>>> There's also no reason why it should maintain dependencies on other
> >>>> parts of Hadoop, if those are separable. How is this solution
> >>>> inadequate?
> >>>>
> >>>> If Chimera is not successful as an independent project or stalls,
> >>>> Hadoop and/or Spark and/or $project will have to reabsorb it as
> >>>> maintainers. Projects have high mortality in early life, and a
> >>>> fight over inheritance/maintenance is something we'd like to avoid.
> >>>> If, on the other hand, it develops enough of a community where it
> >>>> is obviously viable, then we can (and should) break it out as a TLP
> >>>> (as we have before). If other Apache projects take a dependency on
> >>>> Chimera, we're open to adding them to security@hadoop.
> >>>>
> >>>> Unlike Yetus, which was largely rewritten right before it was made
> >>>> into a TLP, security in Hadoop has a complicated pedigree. If
> >>>> Chimera eventually becomes a TLP, it seems fair to include those
> >>>> who work on it while it is a subproject. Declared upfront, that
> >>>> criterion is fairer than any post hoc justification, and will lead
> >>>> to a more accurate account of its community than a subset of the
> >>>> Hadoop PMC/committers that volunteer. -C
> >>>>
> >>>>
> >>>> On Mon, Feb 1, 2016 at 9:29 PM, Chen, Haifeng
> >>>><haifeng.chen@intel.com>
> >>>>wrote:
> >>>>> Thanks to all folks providing feedbacks and participating the
> >>>>>discussions.
> >>>>>
> >>>>> @Owen, do you still have any concerns on going forward in the
> >>>>>direction of Apache Commons (or other options, TLP)?
> >>>>>
> >>>>> Thanks,
> >>>>> Haifeng
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Chen, Haifeng [mailto:haifeng.chen@intel.com]
> >>>>> Sent: Saturday, January 30, 2016 10:52 AM
> >>>>> To: hdfs-dev@hadoop.apache.org
> >>>>> Subject: RE: Hadoop encryption module as Apache Chimera incubator
> >>>>> project
> >>>>>
> >>>>>>> I believe encryption is becoming a core part of Hadoop.
I think
> >>>>>>>that moving core components out of Hadoop is bad from a project
> >>>>>>>management perspective.
> >>>>>
> >>>>>> Although it's certainly true that encryption capabilities (in
> >>>>>>HDFS, YARN, etc.) are becoming core to Hadoop, I don't think
that
> >>>>>>should really influence whether or not the non-Hadoop-specific
> >>>>>>encryption routines should be part of the Hadoop code base, or
> >>>>>>part of the code base of another project that Hadoop depends
on.
> >>>>>>If Chimera had existed as a library hosted at ASF when HDFS
> >>>>>>encryption was first developed, HDFS probably would have just
> >>>>>>added that as a dependency and been done with it. I don't think
we
> >>>>>>would've copy/pasted the code for Chimera into the Hadoop code
base.
> >>>>>
> >>>>> Agree with ATM. I want to also make an additional clarification.
I
> >>>>>agree that the encryption capabilities are becoming core to Hadoop.
> >>>>>While this effort is to put common and shared encryption routines
> >>>>>such as crypto stream implementations into a scope which can be
> >>>>>widely shared across the Apache ecosystem. This doesn't move Hadoop
> >>>>>encryption out of Hadoop (that is not possible).
> >>>>>
> >>>>> Agree if we make it a separate and independent releases project
in
> >>>>>Hadoop takes a step further than the existing approach and solve
> >>>>>some issues (such as libhadoop.so problem). Frankly speaking, I
> >>>>>think it is not the best option we can try. I also expect that an
> >>>>>independent release project within Hadoop core will also complicate
> >>>>>the existing release ideology of Hadoop release.
> >>>>>
> >>>>> Thanks,
> >>>>> Haifeng
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Aaron T. Myers [mailto:atm@cloudera.com]
> >>>>> Sent: Friday, January 29, 2016 9:51 AM
> >>>>> To: hdfs-dev@hadoop.apache.org
> >>>>> Subject: Re: Hadoop encryption module as Apache Chimera incubator
> >>>>> project
> >>>>>
> >>>>> On Wed, Jan 27, 2016 at 11:31 AM, Owen O'Malley
> >>>>><omalley@apache.org>
> >>>>>wrote:
> >>>>>
> >>>>>> I believe encryption is becoming a core part of Hadoop. I think
> >>>>>>that  moving core components out of Hadoop is bad from a project
> >>>>>>management perspective.
> >>>>>>
> >>>>>
> >>>>> Although it's certainly true that encryption capabilities (in
> >>>>>HDFS,  YARN,
> >>>>> etc.) are becoming core to Hadoop, I don't think that should
> >>>>>really influence whether or not the non-Hadoop-specific encryption
> >>>>>routines should be part of the Hadoop code base, or part of the
> >>>>>code base of another project that Hadoop depends on. If Chimera had
> >>>>>existed as a library hosted at ASF when HDFS encryption was first
> >>>>>developed, HDFS probably would have just added that as a dependency
> >>>>>and been done with it. I don't think we would've copy/pasted the
> >>>>>code for Chimera into the Hadoop code base.
> >>>>>
> >>>>>
> >>>>>> To put it another way, a bug in the encryption routines will
> >>>>>> likely become a security problem that security@hadoop needs
to
> hear about.
> >>>>>>
> >>>>> I don't think
> >>>>>> adding a separate project in the middle of that communication
> >>>>>>chain  is a good idea. The same applies to data corruption
> >>>>>>problems, and so on...
> >>>>>>
> >>>>>
> >>>>> Isn't the same true of all the libraries that Hadoop currently
> >>>>>depends upon? If the commons-httpclient library (or commons-codec,
> >>>>>or commons-io, or guava, or...) has a security vulnerability, we
> >>>>>need to know about it so that we can update our dependency to a fixed
> version.
> >>>>>This case doesn't seem materially different than that.
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> > It may be good to keep at generalized place(As in the
> >>>>>> > discussion, we thought that place could be Apache Commons).
> >>>>>>
> >>>>>>
> >>>>>> Apache Commons is a collection of *Java* projects, so Chimera
as
> >>>>>> a JNI-based library isn't a natural fit.
> >>>>>>
> >>>>>
> >>>>> Could very well be that Apache Commons's charter would preclude
> >>>>>Chimera.
> >>>>> You probably know better than I do about that.
> >>>>>
> >>>>>
> >>>>>> Furthermore, Apache Commons doesn't have its own security list
so
> >>>>>> problems will go to the generic security@apache.org.
> >>>>>>
> >>>>>
> >>>>> That seems easy enough to remedy, if they wanted to, and besides
I'm
> >>>>>not sure why that would influence this discussion. In my experience
> >>>>>projects that don't have a separate security@project.a.o mailing
list
> >>>>>tend to just handle security issues on their private@project.a.o
> >>>>>mailing list, which seems fine to me.
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Why do you think that Apache Commons is a better home than Hadoop?
> >>>>>>
> >>>>>
> >>>>> I'm certainly not at all wedded to Apache Commons, that just seemed
> >>>>>like a natural place to put it to me. Could be that a brand new TLP
> >>>>>might make more sense.
> >>>>>
> >>>>> I *do* think that if other non-Hadoop projects want to make use
of
> >>>>>Chimera, which as I understand it is the goal which started this
> >>>>>thread, then Chimera should exist outside of Hadoop so that:
> >>>>>
> >>>>> a) Projects that have nothing to do with Hadoop can just depend
> >>>>>directly on Chimera, which has nothing Hadoop-specific in there.
> >>>>>
> >>>>> b) The Hadoop project doesn't have to export/maintain/concern itself
> >>>>>with yet another publicly-consumed interface.
> >>>>>
> >>>>> c) Chimera can have its own (presumably much faster) release cadence
> >>>>>completely separate from Hadoop.
> >>>>>
> >>>>> --
> >>>>> Aaron T. Myers
> >>>>> Software Engineer, Cloudera
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message