hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <cdoug...@apache.org>
Subject Re: Hadoop encryption module as Apache Chimera incubator project
Date Thu, 04 Feb 2016 22:28:17 GMT
On Thu, Feb 4, 2016 at 12:06 PM, Gangumalla, Uma
<uma.gangumalla@intel.com> wrote:

> [UMA] Ok. Great. You are right. I have cc¹ed to hadoop common. (You mean
> to cc Apache commons as well?)

I meant, if you start a discussion with Apache Commons, please CC
common-dev@hadoop to coordinate.

> [UMA] Right now we plan to have encryption libraries are the only one¹s we
> planned and as we see lot of interest from other projects like spark to
> use them. I see some challenges when we bring lot of code(other common
> codes) into this project is that, they all would have different
> requirements and may be different expected timelines for release etc. Some
> projects may just wanted to use encryption interfaces alone but not all.
> As they are completely independent codes, may be better to scope out
> clearly.

Yes, but even if the artifact is widely consumed, as a TLP it would
need to sustain a community. If the scope is too narrow, then it will
quickly fall into maintenance mode, its contributors will move on, and
it will retire to the attic. Alone, I doubt its viability as a TLP. So
as a first option, donating only this code to Apache Commons would
accomplish some immediate goals in a sustainable forum.

APR has a similar scope. As a second option, that may also be a
reasonable home, particularly if some of the native bits could
integrate with APR.

If the scope is broader, the effort could sustain prolonged
development. The current code is developing a strategy for packing
native libraries on multiple platforms, a capability that, say, the
native compression codecs (AFAIK) still lack. While java.nio is
improving, many projects would benefit from a better, native interface
to the filesystem (e.g., NativeIO). We could avoid duplicating effort
and collaborate on a common library.

As a third option, Hadoop already implements some useful native
libraries, which is why a subproject might be a sound course. That
would enable the subproject to coordinate with Hadoop on migrating its
native functionality to a separable, reusable component, then move to
a TLP when we can rely on it exclusively (if it has a well-defined,
independent community). It could control its release cadence and limit
its dependencies.

Finally, this is beside the point if nobody is interested in doing the
work on such a project. It's rude to pull code out of Hadoop and
donate it to another project so Spark can avoid a dependency, but this
instance seems reasonable to me. -C

[1] https://apr.apache.org/

> On 2/3/16, 6:46 PM, "Chen, Haifeng" <haifeng.chen@intel.com> wrote:
>
>>Thanks Chris.
>>
>>>> I went through the repository, and now understand the reasoning that
>>>>would locate this code in Apache Commons. This isn't proposing to
>>>>extract much of the implementation and it takes none of the
>>>>integration. It's limited to interfaces to crypto libraries and
>>>>streams/configuration.
>>Exactly.
>>
>>>> Chimera would be a boutique TLP, unless we wanted to draw out more of
>>>>the integration and tooling. Is that a goal you're interested in
>>>>pursuing? There's a tension between keeping this focused and including
>>>>enough functionality to make it viable as an independent component.
>>The Chimera goal was for providing useful, common and optimized
>>cryptographic functionalities. I would prefer that it is still focused in
>>this clear scope. Multiple domain requirements will put more challenges
>>and uncertainties in where and how it should go, thus more risk in
>>stalling.
>>
>>>> If the encryption libraries are the only ones you're interested in
>>>>pulling out, then Apache Commons does seem like a better target than a
>>>>separate project.
>>Yes. Just mentioned above, the library will be positioned in
>>cryptographic.
>>
>>
>>Thanks,
>>
>>-----Original Message-----
>>From: Chris Douglas [mailto:cdouglas@apache.org]
>>Sent: Thursday, February 4, 2016 7:26 AM
>>To: hdfs-dev@hadoop.apache.org
>>Subject: Re: Hadoop encryption module as Apache Chimera incubator project
>>
>>I went through the repository, and now understand the reasoning that
>>would locate this code in Apache Commons. This isn't proposing to extract
>>much of the implementation and it takes none of the integration. It's
>>limited to interfaces to crypto libraries and streams/configuration. It
>>might be a reasonable fit for commons-codec, but that's a pretty sparse
>>library and driving the release cadence might be more complicated. It'd
>>be worth discussing on their lists (please also CC common-dev@).
>>
>>Chimera would be a boutique TLP, unless we wanted to draw out more of the
>>integration and tooling. Is that a goal you're interested in pursuing?
>>There's a tension between keeping this focused and including enough
>>functionality to make it viable as an independent component. By way of
>>example, Hadoop's common project requires too many dependencies and
>>carries too much historical baggage for other projects to rely on.
>>I agree with Colin/Steve: we don't want this to grow into another
>>guava-like dependency that creates more work in conflicts than it saves
>>in implementation...
>>
>>Would it make sense to also package some of the compression libraries,
>>and maybe some of the text processing from MapReduce? Evolving some of
>>this code to a common library with few/no dependencies would be generally
>>useful. As a subproject, it could have a broader scope that could evolve
>>into a viable TLP. If the encryption libraries are the only ones you're
>>interested in pulling out, then Apache Commons does seem like a better
>>target than a separate project. -C
>>
>>
>>On Wed, Feb 3, 2016 at 1:49 AM, Chris Douglas <cdouglas@apache.org> wrote:
>>> On Wed, Feb 3, 2016 at 12:48 AM, Gangumalla, Uma
>>> <uma.gangumalla@intel.com> wrote:
>>>>>Standing in the point of shared fundamental piece of code like this,
>>>>>I do think Apache Commons might be the best direction which we can
>>>>>try as the first effort. In this direction, we still need to work
>>>>>with Apache Common community for buying in and accepting the proposal.
>>>> Make sense.
>>>
>>> Makes sense how?
>>>
>>>> For this we should define the independent release cycles for this
>>>> project and it would just place under Hadoop tree if we all conclude
>>>> with this option at the end.
>>>
>>> Yes.
>>>
>>>> [Chris]
>>>>>If Chimera is not successful as an independent project or stalls,
>>>>>Hadoop and/or Spark and/or $project will have to reabsorb it as
>>>>>maintainers.
>>>>>
>>>> I am not so strong on this point. If we assume project would be
>>>> unsuccessful, it can be unsuccessful(less maintained) even under
>>>>hadoop.
>>>> But if other projects depending on this piece then they would get
>>>> less support. Of course right now we feel this piece of code is very
>>>> important and we feel(expect) it can be successful as independent
>>>> project, irrespective of whether it as separate project outside hadoop
>>>>or inside.
>>>> So, I feel this point would not really influence to judge the
>>>>discussion.
>>>
>>> Sure; code can idle anywhere, but that wasn't the point I was after.
>>> You propose to extract code from Hadoop, but if Chimera fails then
>>> what recourse do we have among the other projects taking a dependency
>>> on it? Splitting off another project is feasible, but Chimera should
>>> be sustainable before this PMC can divest itself of responsibility for
>>> security libraries. That's a pretty low bar.
>>>
>>> Bundling the library with the jar is helpful; I've used that before.
>>> It should prefer (updated) libraries from the environment, if
>>> configured. Otherwise it's a pain (or impossible) for ops to patch
>>> security bugs. -C
>>>
>>>>>-----Original Message-----
>>>>>From: Colin P. McCabe [mailto:cmccabe@apache.org]
>>>>>Sent: Wednesday, February 3, 2016 4:56 AM
>>>>>To: hdfs-dev@hadoop.apache.org
>>>>>Subject: Re: Hadoop encryption module as Apache Chimera incubator
>>>>>project
>>>>>
>>>>>It's great to see interest in improving this functionality.  I think
>>>>>Chimera could be successful as an Apache project.  I don't have a
>>>>>strong opinion one way or the other as to whether it belongs as part
>>>>>of Hadoop or separate.
>>>>>
>>>>>I do think there will be some challenges splitting this functionality
>>>>>out into a separate jar, because of the way our CLASSPATH works right
>>>>>now.
>>>>>For example, let's say that Hadoop depends on Chimera 1.2 and Spark
>>>>>depends on Chimera 1.1.  Now Spark jobs have two different versions
>>>>>fighting it out on the classpath, similar to the situation with Guava
>>>>>and other libraries.  Perhaps if Chimera adopts a policy of strong
>>>>>backwards compatibility, we can just always use the latest jar, but
>>>>>it still seems likely that there will be problems.  There are various
>>>>>classpath isolation ideas that could help here, but they are big
>>>>>projects in their own right and we don't have a clear timeline for
>>>>>them.  If this does end up being a separate jar, we may need to shade
>>>>>it to avoid all these issues.
>>>>>
>>>>>Bundling the JNI glue code in the jar itself is an interesting idea,
>>>>>which we have talked about before for libhadoop.so.  It doesn't
>>>>>really have anything to do with the question of TLP vs. non-TLP, of
>>>>>course.
>>>>>We could do that refactoring in Hadoop itself.  The really
>>>>>complicated part of bundling JNI code in a jar is that you need to
>>>>>create jars for every cross product of (JVM version, openssl version,
>>>>>operating system).
>>>>>For example, you have the RHEL6 build for openJDK7 using openssl
>>>>>1.0.1e.
>>>>>If you change any one thing-- say, change openJDK7 to Oracle JDK8,
>>>>>then you might need to rebuild.  And certainly using Ubuntu would be
>>>>>a rebuild.  And so forth.  This kind of clashes with Maven's
>>>>>philosophy of pulling prebuilt jars from the internet.
>>>>>
>>>>>Kai Zheng's question about whether we would bundle openSSL's
>>>>>libraries is a good one.  Given the high rate of new vulnerabilities
>>>>>discovered in that library, it seems like bundling would require
>>>>>Hadoop users and vendors to update very frequently, much more
>>>>>frequently than Hadoop is traditionally updated.  So probably we would
>>>>>not choose to bundle openssl.
>>>>>
>>>>>best,
>>>>>Colin
>>>>>
>>>>>On Tue, Feb 2, 2016 at 12:29 AM, Chris Douglas <cdouglas@apache.org>
>>>>>wrote:
>>>>>> As a subproject of Hadoop, Chimera could maintain its own cadence.
>>>>>> There's also no reason why it should maintain dependencies on other
>>>>>> parts of Hadoop, if those are separable. How is this solution
>>>>>> inadequate?
>>>>>>
>>>>>> If Chimera is not successful as an independent project or stalls,
>>>>>> Hadoop and/or Spark and/or $project will have to reabsorb it as
>>>>>> maintainers. Projects have high mortality in early life, and a
>>>>>> fight over inheritance/maintenance is something we'd like to avoid.
>>>>>> If, on the other hand, it develops enough of a community where it
>>>>>> is obviously viable, then we can (and should) break it out as a TLP
>>>>>> (as we have before). If other Apache projects take a dependency on
>>>>>> Chimera, we're open to adding them to security@hadoop.
>>>>>>
>>>>>> Unlike Yetus, which was largely rewritten right before it was made
>>>>>> into a TLP, security in Hadoop has a complicated pedigree. If
>>>>>> Chimera eventually becomes a TLP, it seems fair to include those
>>>>>> who work on it while it is a subproject. Declared upfront, that
>>>>>> criterion is fairer than any post hoc justification, and will lead
>>>>>> to a more accurate account of its community than a subset of the
>>>>>> Hadoop PMC/committers that volunteer. -C
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 1, 2016 at 9:29 PM, Chen, Haifeng
>>>>>><haifeng.chen@intel.com>
>>>>>>wrote:
>>>>>>> Thanks to all folks providing feedbacks and participating the
>>>>>>>discussions.
>>>>>>>
>>>>>>> @Owen, do you still have any concerns on going forward in the
>>>>>>>direction of Apache Commons (or other options, TLP)?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Haifeng
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Chen, Haifeng [mailto:haifeng.chen@intel.com]
>>>>>>> Sent: Saturday, January 30, 2016 10:52 AM
>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>> Subject: RE: Hadoop encryption module as Apache Chimera incubator
>>>>>>> project
>>>>>>>
>>>>>>>>> I believe encryption is becoming a core part of Hadoop.
I think
>>>>>>>>>that moving core components out of Hadoop is bad from
a project
>>>>>>>>>management perspective.
>>>>>>>
>>>>>>>> Although it's certainly true that encryption capabilities
(in
>>>>>>>>HDFS, YARN, etc.) are becoming core to Hadoop, I don't think
that
>>>>>>>>should really influence whether or not the non-Hadoop-specific
>>>>>>>>encryption routines should be part of the Hadoop code base,
or
>>>>>>>>part of the code base of another project that Hadoop depends
on.
>>>>>>>>If Chimera had existed as a library hosted at ASF when HDFS
>>>>>>>>encryption was first developed, HDFS probably would have just
>>>>>>>>added that as a dependency and been done with it. I don't
think we
>>>>>>>>would've copy/pasted the code for Chimera into the Hadoop
code base.
>>>>>>>
>>>>>>> Agree with ATM. I want to also make an additional clarification.
I
>>>>>>>agree that the encryption capabilities are becoming core to Hadoop.
>>>>>>>While this effort is to put common and shared encryption routines
>>>>>>>such as crypto stream implementations into a scope which can be
>>>>>>>widely shared across the Apache ecosystem. This doesn't move Hadoop
>>>>>>>encryption out of Hadoop (that is not possible).
>>>>>>>
>>>>>>> Agree if we make it a separate and independent releases project
in
>>>>>>>Hadoop takes a step further than the existing approach and solve
>>>>>>>some issues (such as libhadoop.so problem). Frankly speaking,
I
>>>>>>>think it is not the best option we can try. I also expect that
an
>>>>>>>independent release project within Hadoop core will also complicate
>>>>>>>the existing release ideology of Hadoop release.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Haifeng
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Aaron T. Myers [mailto:atm@cloudera.com]
>>>>>>> Sent: Friday, January 29, 2016 9:51 AM
>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>> Subject: Re: Hadoop encryption module as Apache Chimera incubator
>>>>>>> project
>>>>>>>
>>>>>>> On Wed, Jan 27, 2016 at 11:31 AM, Owen O'Malley
>>>>>>><omalley@apache.org>
>>>>>>>wrote:
>>>>>>>
>>>>>>>> I believe encryption is becoming a core part of Hadoop. I
think
>>>>>>>>that  moving core components out of Hadoop is bad from a project
>>>>>>>>management perspective.
>>>>>>>>
>>>>>>>
>>>>>>> Although it's certainly true that encryption capabilities (in
>>>>>>>HDFS,  YARN,
>>>>>>> etc.) are becoming core to Hadoop, I don't think that should
>>>>>>>really influence whether or not the non-Hadoop-specific encryption
>>>>>>>routines should be part of the Hadoop code base, or part of the
>>>>>>>code base of another project that Hadoop depends on. If Chimera
had
>>>>>>>existed as a library hosted at ASF when HDFS encryption was first
>>>>>>>developed, HDFS probably would have just added that as a dependency
>>>>>>>and been done with it. I don't think we would've copy/pasted the
>>>>>>>code for Chimera into the Hadoop code base.
>>>>>>>
>>>>>>>
>>>>>>>> To put it another way, a bug in the encryption routines will
>>>>>>>> likely become a security problem that security@hadoop needs
to
>>>>>>>>hear about.
>>>>>>>>
>>>>>>> I don't think
>>>>>>>> adding a separate project in the middle of that communication
>>>>>>>>chain  is a good idea. The same applies to data corruption
>>>>>>>>problems, and so on...
>>>>>>>>
>>>>>>>
>>>>>>> Isn't the same true of all the libraries that Hadoop currently
>>>>>>>depends upon? If the commons-httpclient library (or commons-codec,
>>>>>>>or commons-io, or guava, or...) has a security vulnerability,
we
>>>>>>>need to know about it so that we can update our dependency to
a
>>>>>>>fixed version.
>>>>>>>This case doesn't seem materially different than that.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> > It may be good to keep at generalized place(As in the
>>>>>>>> > discussion, we thought that place could be Apache Commons).
>>>>>>>>
>>>>>>>>
>>>>>>>> Apache Commons is a collection of *Java* projects, so Chimera
as
>>>>>>>> a JNI-based library isn't a natural fit.
>>>>>>>>
>>>>>>>
>>>>>>> Could very well be that Apache Commons's charter would preclude
>>>>>>>Chimera.
>>>>>>> You probably know better than I do about that.
>>>>>>>
>>>>>>>
>>>>>>>> Furthermore, Apache Commons doesn't have its own security
list so
>>>>>>>> problems will go to the generic security@apache.org.
>>>>>>>>
>>>>>>>
>>>>>>> That seems easy enough to remedy, if they wanted to, and besides
I'm
>>>>>>>not sure why that would influence this discussion. In my experience
>>>>>>>projects that don't have a separate security@project.a.o mailing
list
>>>>>>>tend to just handle security issues on their private@project.a.o
>>>>>>>mailing list, which seems fine to me.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Why do you think that Apache Commons is a better home than
Hadoop?
>>>>>>>>
>>>>>>>
>>>>>>> I'm certainly not at all wedded to Apache Commons, that just
seemed
>>>>>>>like a natural place to put it to me. Could be that a brand new
TLP
>>>>>>>might make more sense.
>>>>>>>
>>>>>>> I *do* think that if other non-Hadoop projects want to make use
of
>>>>>>>Chimera, which as I understand it is the goal which started this
>>>>>>>thread, then Chimera should exist outside of Hadoop so that:
>>>>>>>
>>>>>>> a) Projects that have nothing to do with Hadoop can just depend
>>>>>>>directly on Chimera, which has nothing Hadoop-specific in there.
>>>>>>>
>>>>>>> b) The Hadoop project doesn't have to export/maintain/concern
itself
>>>>>>>with yet another publicly-consumed interface.
>>>>>>>
>>>>>>> c) Chimera can have its own (presumably much faster) release
cadence
>>>>>>>completely separate from Hadoop.
>>>>>>>
>>>>>>> --
>>>>>>> Aaron T. Myers
>>>>>>> Software Engineer, Cloudera
>>>>
>

Mime
View raw message