incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: [PROPOSAL] NiFi for Incubation
Date Fri, 21 Nov 2014 22:16:58 GMT
I didn't make an assertion *about* Accumulo, what I said exactly was:

        "that Accumulo may or may not suffer from "bifurcation" is
debatable"

I assert there are two sides to that argument (I've heard them both)​​,
merely to suggest that Sean's concerns should not be dismissed as they
were.


On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser <elserj@apache.org> wrote:

> I would tend to disagree with your assertion about Accumulo, but this is
> neither the time nor the place to discuss Accumulo's strengths and
> weaknesses.
>
> I'll make the assumption that request to enter incubation is a strong
> intent to grow outside of the project's current home.
>
> +1 (non-binding) for entry
>
>
> Andrew Purtell wrote:
>
>>  From what I understand, that Accumulo may or may not suffer from
>> "bifurcation" is debatable despite this blanket claim otherwise.
>>
>> Regardless, Sean's point is valid, even if only a theoretical concern for
>> NiFi at this point. If the podling is interested in growing outside
>> community I'd be happy to sign on as a mentor to help that happen.
>>
>> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies<bimargulies@gmail.com>
>> wrote:
>>
>>  Sean,
>>>
>>> The precedent of Accumulo is that the govt people and agencies involved
>>> are
>>> ready and able to have their staff collaborate openly in an Apache
>>> community. There's no need to contemplate bifurcation; we have this
>>> proposal because the management recognizes that this collaboration
>>> produces
>>> better stuff that solves more problems than the 'inside the tent'
>>> alternative.
>>>
>>> --benson
>>>
>>>
>>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey<busbey@cloudera.com>
>>> wrote:
>>>
>>>  I'm really excited to see NiFi come to the incubator; it'd be a great
>>>> addition to the ASF.
>>>>
>>>> A few points in the proposal:
>>>>
>>>>  == Initial Goals ==
>>>>>>
>>>>> One of these should be to grow the community outside of the current
>>>>
>>> niche,
>>>
>>>> IMHO.
>>>>
>>>> More on this below under "orphaned projects"
>>>>
>>>>    * Determine and establish a mechanism, possibly including a
>>>>>> sub-project construct, that allows for extensions to the core
>>>>>> application to occur at a pace that differs from the core application
>>>>>> itself.
>>>>>>
>>>>> I don't think the proposal needs to include the "e.g. with
>>>> sub-projects"
>>>> part. Just noting
>>>> that your goals in the incubator are to address the need to have
>>>>
>>> different
>>>
>>>> release cycles
>>>> for core and extensions is sufficient.
>>>>
>>>>
>>>>  === Community ===
>>>>>> Over the past several years, NiFi has developed a strong community
of
>>>>>> both developers and operators within the U.S. government.  We look
>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>
>>>>>>  How much, if any, of this community do you expect to engage via
the
>>>> customary project
>>>> lists once NiFi is established within the ASF? Will the project be able
>>>>
>>> to
>>>
>>>> leverage this
>>>> established group?
>>>>
>>>>
>>>>  === Orphaned Products ===
>>>>>> Risk of orphaning is minimal.  The project user and developer base
is
>>>>>> substantial, growing, and there is already extensive operational
use
>>>>>> of NiFi.
>>>>>>
>>>>> Given that the established base is internal to the U.S. government, I'd
>>>> encourage the
>>>> podling to consider the risk of a bifurcated project should a
>>>> substantial
>>>> outside
>>>> community fail to emerge or if those internal users should fail to
>>>> engage
>>>> with the
>>>> outside community.
>>>>
>>>> You cover a related issue in your "Homogenous Developers" section. But I
>>>> think
>>>> building on the "Community" section of the current state to call this
>>>> out
>>>> as an
>>>> independent issue is worthwhile.
>>>>
>>>>
>>>>  possible.  This environment includes widely accessible source code
>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>> documentation. We also encourage contributions and frequent debate
and
>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>> and in-person meet-ups.
>>>>>>
>>>>> Do you anticipate any difficulties moving these established
>>>> communication
>>>> mechanisms to ASF public lists?
>>>>
>>>>  === Documentation ===
>>>>>> At this time there is no NiFi documentation on the web.  However,
we
>>>>>> have extensive documentation included within the application that
>>>>>> details usage of the many functions.  We will be rapidly expanding
the
>>>>>> available documentation to cover things like installation, developer
>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>
>>>>> I love projects that start with documentation. :)
>>>>
>>>> I don't think the proposal needs to include that the documentation will
>>>>
>>> be
>>>
>>>> posted
>>>> to the NiFi wiki, since that's an implementation detail. Just say "this
>>>> documentation
>>>> will be made available via the NiFi project's use of incubator infra."
>>>>
>>>> (I'll save detail for the eventual dev@ list, but you should strongly
>>>> consider not
>>>> using the wiki to host this documentation.)
>>>>
>>>> -Sean
>>>>
>>>> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland<brock@cloudera.com>
>>>>
>>> wrote:
>>>
>>>> Hi Joe,
>>>>>
>>>>> I know you've done a tremendous amount of work to make this happen so
I
>>>>>
>>>> am
>>>>
>>>>> extremely happy this is *finally* making it's way to the incubator!
>>>>>
>>>>> I look forward to helping in anyway I can.
>>>>>
>>>>> Cheers!
>>>>> Brock
>>>>>
>>>>> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)<
>>>>> chris.a.mattmann@jpl.nasa.gov>  wrote:
>>>>>
>>>>>  This is *fan freakin¹ tastic* Sounds like an awesome  project and
>>>>>> glad to hear a relationship to Tika! Awesome to see more government
>>>>>> projects coming into the ASF!
>>>>>>
>>>>>> you already have a great set of mentors and I don¹t really have
more
>>>>>> time on my plate, but really happy and will try and monitor and help
>>>>>> on the lists.
>>>>>>
>>>>>> Cheers!
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Chief Architect
>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Joe Witt<joe.witt@gmail.com>
>>>>>> Reply-To: "general@incubator.apache.org"<
>>>>>>
>>>>> general@incubator.apache.org
>>>
>>>> Date: Thursday, November 20, 2014 at 3:02 AM
>>>>>> To: "general@incubator.apache.org"<general@incubator.apache.org>
>>>>>> Subject: [PROPOSAL] NiFi for Incubation
>>>>>>
>>>>>>  Hello,
>>>>>>>
>>>>>>>
>>>>>>> I would like to propose NiFi as an Apache Incubator Project.
>>>>>>>
>>>>>>> In addition to the copy provided below the Wiki version of the
>>>>>>> proposal can be found here:
>>>>>>> http://wiki.apache.org/incubator/NiFiProposal
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Joe
>>>>>>>
>>>>>>>
>>>>>>> = NiFi Proposal =
>>>>>>>
>>>>>>> == Abstract ==
>>>>>>> NiFi is a dataflow system based on the concepts of flow-based
>>>>>>>
>>>>>> programming.
>>>>>
>>>>>> == Proposal ==
>>>>>>> NiFi supports powerful and scalable directed graphs of data routing,
>>>>>>> transformation, and system mediation logic.  Some of the high-level
>>>>>>> capabilities and objectives of NiFi include:
>>>>>>>   * Web-based user interface for seamless experience between
design,
>>>>>>> control, feedback, and monitoring of data flows
>>>>>>>   * Highly configurable along several dimensions of quality of
>>>>>>>
>>>>>> service
>>>
>>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>>>>>> high throughput, and priority based queuing
>>>>>>>   * Fine-grained data provenance for all data received, forked,
>>>>>>> joined, cloned, modified, sent, and ultimately dropped as data
>>>>>>>
>>>>>> reaches
>>>
>>>> its configured end-state
>>>>>>>   * Component-based extension model along well defined interfaces
>>>>>>> enabling rapid development and effective testing
>>>>>>>
>>>>>>> == Background ==
>>>>>>> Reliable and effective dataflow between systems can be difficult
>>>>>>> whether you're running scripts on a laptop or have a massive
>>>>>>> distributed computing system operated by numerous teams and
>>>>>>> organizations.  As the volume and rate of data grows and as the
>>>>>>>
>>>>>> number
>>>
>>>> of systems, protocols, and formats increase and evolve so too does
>>>>>>>
>>>>>> the
>>>
>>>> complexity and need for greater insight and agility.  These are the
>>>>>>> dataflow challenges that NiFi was built to tackle.
>>>>>>>
>>>>>>> NiFi is designed in a manner consistent with the core concepts
>>>>>>> described in flow-based programming as originally documented
by J.
>>>>>>> Paul Morrison in the 1970s.  This model lends itself well to
visual
>>>>>>> diagramming, concurrency, componentization, testing, and reuse.
 In
>>>>>>> addition to staying close to the fundamentals of flow-based
>>>>>>> programming, NiFi provides integration system specific features
such
>>>>>>> as: guaranteed delivery; back pressure; ability to gracefully
handle
>>>>>>> backlogs and data surges; and an operator interface that enables
>>>>>>> on-the-fly data flow generation, modification, and observation.
>>>>>>>
>>>>>>> == Rationale ==
>>>>>>> NiFi provides a reliable, scalable, manageable and accountable
>>>>>>> platform for developers and technical staff to create and evolve
>>>>>>> powerful data flows.  Such a system is useful in many contexts
>>>>>>> including large-scale enterprise integration, interaction with
cloud
>>>>>>> services and frameworks, business to business, intra-departmental,
>>>>>>>
>>>>>> and
>>>
>>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>>>>>> Foundation (ASF) family as it depends on numerous ASF projects
and
>>>>>>> integrates with several others.  We also anticipate developing
>>>>>>> extensions for several other ASF projects such as Cassandra,
Kafka,
>>>>>>> and Storm in the near future.
>>>>>>>
>>>>>>> == Initial Goals ==
>>>>>>>   * Ensure all dependencies are compliant with Apache License
>>>>>>>
>>>>>> version
>>>
>>>> 2.0 and all that all code and documentation artifacts have the
>>>>>>>
>>>>>> correct
>>>
>>>> Apache licensing markings and notice.
>>>>>>>   * Establish a formal release process and schedule, allowing
for
>>>>>>> dependable release cycles in a manner consistent with the Apache
>>>>>>> development process.
>>>>>>>   * Determine and establish a mechanism, possibly including a
>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>> application to occur at a pace that differs from the core
>>>>>>>
>>>>>> application
>>>
>>>> itself.
>>>>>>>
>>>>>>> == Current Status ==
>>>>>>> === Meritocracy ===
>>>>>>> An integration platform is only as good as its ability to integrate
>>>>>>> systems in a reliable, timely, and repeatable manner.  The same
can
>>>>>>>
>>>>>> be
>>>
>>>> said of its ability to attract talent and a variety of perspectives
>>>>>>>
>>>>>> as
>>>
>>>> integration systems by their nature are always evolving.  We will
>>>>>>> actively seek help and encourage promotion of influence in the
>>>>>>>
>>>>>> project
>>>
>>>> through meritocracy.
>>>>>>>
>>>>>>> === Community ===
>>>>>>> Over the past several years, NiFi has developed a strong community
>>>>>>>
>>>>>> of
>>>
>>>> both developers and operators within the U.S. government.  We look
>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>
>>>>>>> === Core Developers ===
>>>>>>> The initial core developers are employed by the National Security
>>>>>>> Agency and defense contractors.  We will work to grow the community
>>>>>>> among a more diverse set of developers and industries.
>>>>>>>
>>>>>>> === Alignment ===
>>>>>>>
>>>>>> > From its inception, NiFi was developed with an open source
>>>>>>
>>>>> philosophy
>>>
>>>> in mind and with the hopes of eventually being truly open sourced.
>>>>>>> The Apache way is consistent with the approach we have taken
to
>>>>>>>
>>>>>> date.
>>>
>>>> The ASF clearly provides a mature and effective environment for
>>>>>>> successful development as is evident across the spectrum of
>>>>>>>
>>>>>> well-known
>>>
>>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>>>>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>>>>>> HttpClient, Jakarta and Maven.  We also anticipate extensions
and
>>>>>>> dependencies with several more ASF projects, including Accumulo,
>>>>>>>
>>>>>> Avro,
>>>
>>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>>>>>> This existing alignment with Apache and the desired community
makes
>>>>>>> the Apache Incubator a good fit for NiFi.
>>>>>>>
>>>>>>> == Known Risks ==
>>>>>>> === Orphaned Products ===
>>>>>>> Risk of orphaning is minimal.  The project user and developer
base
>>>>>>>
>>>>>> is
>>>
>>>> substantial, growing, and there is already extensive operational use
>>>>>>> of NiFi.
>>>>>>>
>>>>>>> === Inexperience with Open Source ===
>>>>>>> The initial committers to NiFi have limited experience with true
>>>>>>>
>>>>>> open
>>>
>>>> source software development.  However, despite the project origins
>>>>>>> being from closed source development we have modelled our behavior
>>>>>>>
>>>>>> and
>>>
>>>> community development on The Apache Way to the greatest extent
>>>>>>> possible.  This environment includes widely accessible source
code
>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>
>>>>>> and
>>>
>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>> and in-person meet-ups.  We are committed to the ideals of open
>>>>>>>
>>>>>> source
>>>
>>>> software and will eagerly seek out mentors and sponsors who can help
>>>>>>> us quickly come up to speed.
>>>>>>>
>>>>>>> === Homogenous Developers ===
>>>>>>> The initial committers of NiFi come from a limited set of entities
>>>>>>> though we are committed to recruiting and developing additional
>>>>>>> committers from a broad spectrum of industries and backgrounds.
>>>>>>>
>>>>>>> === Reliance on Salaried Developers ===
>>>>>>> We expect NiFi development to continue on salaried time and through
>>>>>>> volunteer time.  The initial committers are paid by their employers
>>>>>>>
>>>>>> to
>>>
>>>> contribute to this project.  We are committed to developing and
>>>>>>> recruiting participation from developers both salaried and
>>>>>>> non-salaried.
>>>>>>>
>>>>>>> === Relationship with other Apache Projects ===
>>>>>>> As described in the alignment section, NiFi is already heavily
>>>>>>> dependent on other ASF projects and we anticipate further dependence
>>>>>>> and integration with new and emerging projects in the Apache
family.
>>>>>>>
>>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>>> We respect the laudable Apache brand and that is certainly a
factor
>>>>>>>
>>>>>> in
>>>
>>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>>>>>> provide a great deal of energy and capability to the ASF through
>>>>>>>
>>>>>> this
>>>
>>>> project.  We will be sensitive to and respectful of any overuse of
>>>>>>>
>>>>>> the
>>>
>>>> Apache brand and ensure our focus remains on how we benefit the
>>>>>>>
>>>>>> Apache
>>>
>>>> community.
>>>>>>>
>>>>>>> === Documentation ===
>>>>>>> At this time there is no NiFi documentation on the web.  However,
we
>>>>>>> have extensive documentation included within the application
that
>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>
>>>>>> the
>>>
>>>> available documentation to cover things like installation, developer
>>>>>>> guide, frequently asked questions, best practices, and more.
 This
>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>
>>>>>>> == Initial Source ==
>>>>>>> NiFi has been in active development since late 2006 with
>>>>>>>
>>>>>> contributions
>>>
>>>> >from dozens of developers and feedback from hundreds of users and
>>>>>>
>>>>>>> developers.  The core codebase is written in Java and includes
>>>>>>> detailed Javadocs and feature documentation.
>>>>>>>
>>>>>>> == Source and Intellectual Property Submission ==
>>>>>>> Previously referred to as Niagarafiles, the NiFi code and
>>>>>>> documentation materials will be submitted by the National Security
>>>>>>> Agency.  NiFi has been developed by a mix of government employees
>>>>>>>
>>>>>> and
>>>
>>>> private companies under government contract.  Material developed by
>>>>>>> the government employees is in the public domain and no U.S.
>>>>>>>
>>>>>> copyright
>>>
>>>> exists in works of the federal government.  For the contractor
>>>>>>> developed material in the initial submission, the U.S. Government
>>>>>>>
>>>>>> has
>>>
>>>> sufficient authority to open source per DFARS 252.227-7014.  NSA
>>>>>>>
>>>>>> has
>>>
>>>> submitted the Software Grant Agreement and Corporate Contributor
>>>>>>> License Agreement to the Apache Software Foundation.
>>>>>>>
>>>>>>> == External Dependencies ==
>>>>>>> We have at least one dependency on an LGPL library which we will
>>>>>>> promptly address.  Otherwise, we believe all current dependencies
>>>>>>>
>>>>>> are
>>>
>>>> compatible with the ASF guidelines.  Our dependency licenses come
>>>>>>>
>>>>>> from
>>>
>>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>>>>>> Eclipse Public v1, MIT, CDDL v1.
>>>>>>>
>>>>>>> == Cryptography ==
>>>>>>> Consistent with http://www.apache.org/licenses/exports/ we believe
>>>>>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>>>>>> cryptographic algorithms but is designed to use algorithms provided
>>>>>>>
>>>>>> by
>>>
>>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>>>>>> These cryptographic algorithm providers are used to support SSL,
>>>>>>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>>>>>>> In the event that it becomes necessary we will engage with
>>>>>>>
>>>>>> appropriate
>>>
>>>> Apache members to ensure we file any necessary paperwork or
>>>>>>>
>>>>>> clarified
>>>
>>>> any cryptographic export license concerns.
>>>>>>>
>>>>>>> == Required Resources ==
>>>>>>> === Mailing Lists ===
>>>>>>>   * user@nifi.incubator.apache.org
>>>>>>>   * dev@nifi.incubator.apache.org
>>>>>>>   * private@nifi.incubator.apache.org
>>>>>>>   * commits@nifi.incubator.apache.org
>>>>>>>
>>>>>>> === Source Control ===
>>>>>>> NiFi requests use of Git for source control
>>>>>>> (git://git.apache.org/nifi.git).  We request a writeable Git
repo
>>>>>>>
>>>>>> for
>>>
>>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>>>>>> sponsor Benson Margulies (bimargulies) to assist with creating
the
>>>>>>> INFRA ticket for this.
>>>>>>>
>>>>>>> === Issue Tracking ===
>>>>>>> JIRA NiFi (NIFI)
>>>>>>>
>>>>>>> === Initial Committers ===
>>>>>>>   * Brandon Devries<brandon.devries at gmail dot com>
>>>>>>>   * Matt Gilman<matt.c.gilman at gmail dot com>
>>>>>>>   * Tony Kurc<trkurc at gmail dot com>
>>>>>>>   * Mark Payne<markap14 at hotmail dot com>
>>>>>>>   * Adam Taft<adam at adamtaft dot com>
>>>>>>>   * Joseph Witt<joewitt at gmail dot com>
>>>>>>>
>>>>>>> === Affiliations ===
>>>>>>>   * Brandon Devries (Requitest, Inc.)
>>>>>>>   * Matt Gilman (Raytheon)
>>>>>>>   * Tony Kurc (National Security Agency)
>>>>>>>   * Mark Payne (Sotera Defense Solutions, Inc.)
>>>>>>>   * Adam Taft (Requitest, Inc.)
>>>>>>>   * Joseph Witt (National Security Agency)
>>>>>>>
>>>>>>> == Sponsors ==
>>>>>>> === Champion ===
>>>>>>>   * Benson Margulies (Basis Technology)<bimargulies at apache
dot
>>>>>>>
>>>>>> org>
>>>>
>>>>> === Nominated Mentors ===
>>>>>>>   * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>>>>>   * Brock Noland (Cloudera)<brock at apache dot org>
>>>>>>>   * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>>>>>
>>>>>>> === Sponsoring Entity ===
>>>>>>> We request the Apache Incubator to sponsor this project.
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Sean
>>>>
>>>>
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message