incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: [PROPOSAL] NiFi for Incubation
Date Fri, 21 Nov 2014 21:54:21 GMT
>From what I understand, that Accumulo may or may not suffer from
"bifurcation" is debatable despite this blanket claim otherwise.

Regardless, Sean's point is valid, even if only a theoretical concern for
NiFi at this point. If the podling is interested in growing outside
community I'd be happy to sign on as a mentor to help that happen.

On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies <bimargulies@gmail.com>
wrote:

> Sean,
>
> The precedent of Accumulo is that the govt people and agencies involved are
> ready and able to have their staff collaborate openly in an Apache
> community. There's no need to contemplate bifurcation; we have this
> proposal because the management recognizes that this collaboration produces
> better stuff that solves more problems than the 'inside the tent'
> alternative.
>
> --benson
>
>
> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey <busbey@cloudera.com> wrote:
>
> > I'm really excited to see NiFi come to the incubator; it'd be a great
> > addition to the ASF.
> >
> > A few points in the proposal:
> >
> > > >== Initial Goals ==
> >
> > One of these should be to grow the community outside of the current
> niche,
> > IMHO.
> >
> > More on this below under "orphaned projects"
> >
> > > >  * Determine and establish a mechanism, possibly including a
> > > >sub-project construct, that allows for extensions to the core
> > > >application to occur at a pace that differs from the core application
> > > >itself.
> >
> > I don't think the proposal needs to include the "e.g. with sub-projects"
> > part. Just noting
> > that your goals in the incubator are to address the need to have
> different
> > release cycles
> > for core and extensions is sufficient.
> >
> >
> > > >=== Community ===
> > > >Over the past several years, NiFi has developed a strong community of
> > > >both developers and operators within the U.S. government.  We look
> > > >forward to helping grow this to a broader base of industries.
> > > >
> >
> > How much, if any, of this community do you expect to engage via the
> > customary project
> > lists once NiFi is established within the ASF? Will the project be able
> to
> > leverage this
> > established group?
> >
> >
> > > >=== Orphaned Products ===
> > > >Risk of orphaning is minimal.  The project user and developer base is
> > > >substantial, growing, and there is already extensive operational use
> > > >of NiFi.
> >
> > Given that the established base is internal to the U.S. government, I'd
> > encourage the
> > podling to consider the risk of a bifurcated project should a substantial
> > outside
> > community fail to emerge or if those internal users should fail to engage
> > with the
> > outside community.
> >
> > You cover a related issue in your "Homogenous Developers" section. But I
> > think
> > building on the "Community" section of the current state to call this out
> > as an
> > independent issue is worthwhile.
> >
> >
> > > >possible.  This environment includes widely accessible source code
> > > >repositories, published artifacts, ticket tracking, and extensive
> > > >documentation. We also encourage contributions and frequent debate and
> > > >hold regular, collaborative discussions through e-mail, chat rooms,
> > > >and in-person meet-ups.
> >
> > Do you anticipate any difficulties moving these established communication
> > mechanisms to ASF public lists?
> >
> > > >=== Documentation ===
> > > >At this time there is no NiFi documentation on the web.  However, we
> > > >have extensive documentation included within the application that
> > > >details usage of the many functions.  We will be rapidly expanding the
> > > >available documentation to cover things like installation, developer
> > > >guide, frequently asked questions, best practices, and more.  This
> > > >documentation will be posted to the NiFi wiki at apache.org.
> >
> > I love projects that start with documentation. :)
> >
> > I don't think the proposal needs to include that the documentation will
> be
> > posted
> > to the NiFi wiki, since that's an implementation detail. Just say "this
> > documentation
> > will be made available via the NiFi project's use of incubator infra."
> >
> > (I'll save detail for the eventual dev@ list, but you should strongly
> > consider not
> > using the wiki to host this documentation.)
> >
> > -Sean
> >
> > On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland <brock@cloudera.com>
> wrote:
> >
> > > Hi Joe,
> > >
> > > I know you've done a tremendous amount of work to make this happen so I
> > am
> > > extremely happy this is *finally* making it's way to the incubator!
> > >
> > > I look forward to helping in anyway I can.
> > >
> > > Cheers!
> > > Brock
> > >
> > > On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > >
> > > > This is *fan freakinĀ¹ tastic* Sounds like an awesome  project and
> > > > glad to hear a relationship to Tika! Awesome to see more government
> > > > projects coming into the ASF!
> > > >
> > > > you already have a great set of mentors and I donĀ¹t really have more
> > > > time on my plate, but really happy and will try and monitor and help
> > > > on the lists.
> > > >
> > > > Cheers!
> > > >
> > > > Chris
> > > >
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > Chris Mattmann, Ph.D.
> > > > Chief Architect
> > > > Instrument Software and Science Data Systems Section (398)
> > > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > Office: 168-519, Mailstop: 168-527
> > > > Email: chris.a.mattmann@nasa.gov
> > > > WWW:  http://sunset.usc.edu/~mattmann/
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > Adjunct Associate Professor, Computer Science Department
> > > > University of Southern California, Los Angeles, CA 90089 USA
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Joe Witt <joe.witt@gmail.com>
> > > > Reply-To: "general@incubator.apache.org" <
> general@incubator.apache.org
> > >
> > > > Date: Thursday, November 20, 2014 at 3:02 AM
> > > > To: "general@incubator.apache.org" <general@incubator.apache.org>
> > > > Subject: [PROPOSAL] NiFi for Incubation
> > > >
> > > > >Hello,
> > > > >
> > > > >
> > > > >I would like to propose NiFi as an Apache Incubator Project.
> > > > >
> > > > >In addition to the copy provided below the Wiki version of the
> > > > >proposal can be found here:
> > > > >http://wiki.apache.org/incubator/NiFiProposal
> > > > >
> > > > >Thanks
> > > > >
> > > > >Joe
> > > > >
> > > > >
> > > > >= NiFi Proposal =
> > > > >
> > > > >== Abstract ==
> > > > >NiFi is a dataflow system based on the concepts of flow-based
> > > programming.
> > > > >
> > > > >== Proposal ==
> > > > >NiFi supports powerful and scalable directed graphs of data routing,
> > > > >transformation, and system mediation logic.  Some of the high-level
> > > > >capabilities and objectives of NiFi include:
> > > > >  * Web-based user interface for seamless experience between design,
> > > > >control, feedback, and monitoring of data flows
> > > > >  * Highly configurable along several dimensions of quality of
> service
> > > > >such as loss tolerant versus guaranteed delivery, low latency versus
> > > > >high throughput, and priority based queuing
> > > > >  * Fine-grained data provenance for all data received, forked,
> > > > >joined, cloned, modified, sent, and ultimately dropped as data
> reaches
> > > > >its configured end-state
> > > > >  * Component-based extension model along well defined interfaces
> > > > >enabling rapid development and effective testing
> > > > >
> > > > >== Background ==
> > > > >Reliable and effective dataflow between systems can be difficult
> > > > >whether you're running scripts on a laptop or have a massive
> > > > >distributed computing system operated by numerous teams and
> > > > >organizations.  As the volume and rate of data grows and as the
> number
> > > > >of systems, protocols, and formats increase and evolve so too does
> the
> > > > >complexity and need for greater insight and agility.  These are the
> > > > >dataflow challenges that NiFi was built to tackle.
> > > > >
> > > > >NiFi is designed in a manner consistent with the core concepts
> > > > >described in flow-based programming as originally documented by J.
> > > > >Paul Morrison in the 1970s.  This model lends itself well to visual
> > > > >diagramming, concurrency, componentization, testing, and reuse.  In
> > > > >addition to staying close to the fundamentals of flow-based
> > > > >programming, NiFi provides integration system specific features such
> > > > >as: guaranteed delivery; back pressure; ability to gracefully handle
> > > > >backlogs and data surges; and an operator interface that enables
> > > > >on-the-fly data flow generation, modification, and observation.
> > > > >
> > > > >== Rationale ==
> > > > >NiFi provides a reliable, scalable, manageable and accountable
> > > > >platform for developers and technical staff to create and evolve
> > > > >powerful data flows.  Such a system is useful in many contexts
> > > > >including large-scale enterprise integration, interaction with cloud
> > > > >services and frameworks, business to business, intra-departmental,
> and
> > > > >inter-departmental flows.  NiFi fits well within the Apache Software
> > > > >Foundation (ASF) family as it depends on numerous ASF projects and
> > > > >integrates with several others.  We also anticipate developing
> > > > >extensions for several other ASF projects such as Cassandra, Kafka,
> > > > >and Storm in the near future.
> > > > >
> > > > >== Initial Goals ==
> > > > >  * Ensure all dependencies are compliant with Apache License
> version
> > > > >2.0 and all that all code and documentation artifacts have the
> correct
> > > > >Apache licensing markings and notice.
> > > > >  * Establish a formal release process and schedule, allowing for
> > > > >dependable release cycles in a manner consistent with the Apache
> > > > >development process.
> > > > >  * Determine and establish a mechanism, possibly including a
> > > > >sub-project construct, that allows for extensions to the core
> > > > >application to occur at a pace that differs from the core
> application
> > > > >itself.
> > > > >
> > > > >== Current Status ==
> > > > >=== Meritocracy ===
> > > > >An integration platform is only as good as its ability to integrate
> > > > >systems in a reliable, timely, and repeatable manner.  The same can
> be
> > > > >said of its ability to attract talent and a variety of perspectives
> as
> > > > >integration systems by their nature are always evolving.  We will
> > > > >actively seek help and encourage promotion of influence in the
> project
> > > > >through meritocracy.
> > > > >
> > > > >=== Community ===
> > > > >Over the past several years, NiFi has developed a strong community
> of
> > > > >both developers and operators within the U.S. government.  We look
> > > > >forward to helping grow this to a broader base of industries.
> > > > >
> > > > >=== Core Developers ===
> > > > >The initial core developers are employed by the National Security
> > > > >Agency and defense contractors.  We will work to grow the community
> > > > >among a more diverse set of developers and industries.
> > > > >
> > > > >=== Alignment ===
> > > > >From its inception, NiFi was developed with an open source
> philosophy
> > > > >in mind and with the hopes of eventually being truly open sourced.
> > > > >The Apache way is consistent with the approach we have taken to
> date.
> > > > >The ASF clearly provides a mature and effective environment for
> > > > >successful development as is evident across the spectrum of
> well-known
> > > > >projects.  Further, NiFi depends on numerous ASF libraries and
> > > > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> > > > >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> > > > >dependencies with several more ASF projects, including Accumulo,
> Avro,
> > > > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> > > > >This existing alignment with Apache and the desired community makes
> > > > >the Apache Incubator a good fit for NiFi.
> > > > >
> > > > >== Known Risks ==
> > > > >=== Orphaned Products ===
> > > > >Risk of orphaning is minimal.  The project user and developer base
> is
> > > > >substantial, growing, and there is already extensive operational use
> > > > >of NiFi.
> > > > >
> > > > >=== Inexperience with Open Source ===
> > > > >The initial committers to NiFi have limited experience with true
> open
> > > > >source software development.  However, despite the project origins
> > > > >being from closed source development we have modelled our behavior
> and
> > > > >community development on The Apache Way to the greatest extent
> > > > >possible.  This environment includes widely accessible source code
> > > > >repositories, published artifacts, ticket tracking, and extensive
> > > > >documentation. We also encourage contributions and frequent debate
> and
> > > > >hold regular, collaborative discussions through e-mail, chat rooms,
> > > > >and in-person meet-ups.  We are committed to the ideals of open
> source
> > > > >software and will eagerly seek out mentors and sponsors who can help
> > > > >us quickly come up to speed.
> > > > >
> > > > >=== Homogenous Developers ===
> > > > >The initial committers of NiFi come from a limited set of entities
> > > > >though we are committed to recruiting and developing additional
> > > > >committers from a broad spectrum of industries and backgrounds.
> > > > >
> > > > >=== Reliance on Salaried Developers ===
> > > > >We expect NiFi development to continue on salaried time and through
> > > > >volunteer time.  The initial committers are paid by their employers
> to
> > > > >contribute to this project.  We are committed to developing and
> > > > >recruiting participation from developers both salaried and
> > > > >non-salaried.
> > > > >
> > > > >=== Relationship with other Apache Projects ===
> > > > >As described in the alignment section, NiFi is already heavily
> > > > >dependent on other ASF projects and we anticipate further dependence
> > > > >and integration with new and emerging projects in the Apache family.
> > > > >
> > > > >=== An Excessive Fascination with the Apache Brand ===
> > > > >We respect the laudable Apache brand and that is certainly a factor
> in
> > > > >the decision to propose NiFi for the Apache Incubator.  We intend
to
> > > > >provide a great deal of energy and capability to the ASF through
> this
> > > > >project.  We will be sensitive to and respectful of any overuse of
> the
> > > > >Apache brand and ensure our focus remains on how we benefit the
> Apache
> > > > >community.
> > > > >
> > > > >=== Documentation ===
> > > > >At this time there is no NiFi documentation on the web.  However,
we
> > > > >have extensive documentation included within the application that
> > > > >details usage of the many functions.  We will be rapidly expanding
> the
> > > > >available documentation to cover things like installation, developer
> > > > >guide, frequently asked questions, best practices, and more.  This
> > > > >documentation will be posted to the NiFi wiki at apache.org.
> > > > >
> > > > >== Initial Source ==
> > > > >NiFi has been in active development since late 2006 with
> contributions
> > > > >from dozens of developers and feedback from hundreds of users and
> > > > >developers.  The core codebase is written in Java and includes
> > > > >detailed Javadocs and feature documentation.
> > > > >
> > > > >== Source and Intellectual Property Submission ==
> > > > >Previously referred to as Niagarafiles, the NiFi code and
> > > > >documentation materials will be submitted by the National Security
> > > > >Agency.  NiFi has been developed by a mix of government employees
> and
> > > > >private companies under government contract.  Material developed by
> > > > >the government employees is in the public domain and no U.S.
> copyright
> > > > >exists in works of the federal government.  For the contractor
> > > > >developed material in the initial submission, the U.S. Government
> has
> > > > >sufficient authority to open source per DFARS 252.227-7014.  NSA
> has
> > > > >submitted the Software Grant Agreement and Corporate Contributor
> > > > >License Agreement to the Apache Software Foundation.
> > > > >
> > > > >== External Dependencies ==
> > > > >We have at least one dependency on an LGPL library which we will
> > > > >promptly address.  Otherwise, we believe all current dependencies
> are
> > > > >compatible with the ASF guidelines.  Our dependency licenses come
> from
> > > > >the following license styles:  Apache v 2.0, BSD, Public Domain,
> > > > >Eclipse Public v1, MIT, CDDL v1.
> > > > >
> > > > >== Cryptography ==
> > > > >Consistent with http://www.apache.org/licenses/exports/ we believe
> > > > >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> > > > >cryptographic algorithms but is designed to use algorithms provided
> by
> > > > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> > > > >These cryptographic algorithm providers are used to support SSL,
> > > > >SSH/SFTP, and the encryption and decryption of sensitive properties.
> > > > >In the event that it becomes necessary we will engage with
> appropriate
> > > > >Apache members to ensure we file any necessary paperwork or
> clarified
> > > > >any cryptographic export license concerns.
> > > > >
> > > > >== Required Resources ==
> > > > >=== Mailing Lists ===
> > > > >  * user@nifi.incubator.apache.org
> > > > >  * dev@nifi.incubator.apache.org
> > > > >  * private@nifi.incubator.apache.org
> > > > >  * commits@nifi.incubator.apache.org
> > > > >
> > > > >=== Source Control ===
> > > > >NiFi requests use of Git for source control
> > > > >(git://git.apache.org/nifi.git).  We request a writeable Git repo
> for
> > > > >NiFi with mirroring to be setup to Github through INFRA.  We request
> > > > >sponsor Benson Margulies (bimargulies) to assist with creating the
> > > > >INFRA ticket for this.
> > > > >
> > > > >=== Issue Tracking ===
> > > > >JIRA NiFi (NIFI)
> > > > >
> > > > >=== Initial Committers ===
> > > > >  * Brandon Devries <brandon.devries at gmail dot com>
> > > > >  * Matt Gilman <matt.c.gilman at gmail dot com>
> > > > >  * Tony Kurc <trkurc at gmail dot com>
> > > > >  * Mark Payne <markap14 at hotmail dot com>
> > > > >  * Adam Taft <adam at adamtaft dot com>
> > > > >  * Joseph Witt <joewitt at gmail dot com>
> > > > >
> > > > >=== Affiliations ===
> > > > >  * Brandon Devries (Requitest, Inc.)
> > > > >  * Matt Gilman (Raytheon)
> > > > >  * Tony Kurc (National Security Agency)
> > > > >  * Mark Payne (Sotera Defense Solutions, Inc.)
> > > > >  * Adam Taft (Requitest, Inc.)
> > > > >  * Joseph Witt (National Security Agency)
> > > > >
> > > > >== Sponsors ==
> > > > >=== Champion ===
> > > > >  * Benson Margulies (Basis Technology) <bimargulies at apache
dot
> > org>
> > > > >
> > > > >=== Nominated Mentors ===
> > > > >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> > > > >  * Brock Noland (Cloudera) <brock at apache dot org>
> > > > >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> > > > >
> > > > >=== Sponsoring Entity ===
> > > > >We request the Apache Incubator to sponsor this project.
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sean
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message