incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <>
Subject Re: [PROPOSAL] NiFi for Incubation
Date Fri, 21 Nov 2014 12:20:47 GMT

Thank you for offering to be a mentor.  The proposal has been updated.

Arvind Prabhakar (StreamSets) <arvind at apache dot org>

One of the comments we have received thus far is a potential concern of
whether multiple communities/projects can exist in similar problem areas.
Your background in Flume which solves similar problems to NiFi is a good
sign that multiple projects and communities can co-exist within the space
and even provide complimentary capability.


On Thu, Nov 20, 2014 at 9:20 PM, Joe Witt <> wrote:

> Josh,
> Really appreciate it and have updated the proposal.
> Thanks
> Joe
> On Thu, Nov 20, 2014 at 9:35 AM, Josh Elser <> wrote:
>> Very exciting stuff!
>> Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as
>> a mentor. If so, I'll submit an application to join the IPMC and we can go
>> from there.
>> - Josh
>> Joe Witt wrote:
>>> Hello,
>>> I would like to propose NiFi as an Apache Incubator Project.
>>> In addition to the copy provided below the Wiki version of the
>>> proposal can be found here:
>>> Thanks
>>> Joe
>>> = NiFi Proposal =
>>> == Abstract ==
>>> NiFi is a dataflow system based on the concepts of flow-based
>>> programming.
>>> == Proposal ==
>>> NiFi supports powerful and scalable directed graphs of data routing,
>>> transformation, and system mediation logic.  Some of the high-level
>>> capabilities and objectives of NiFi include:
>>>    * Web-based user interface for seamless experience between design,
>>> control, feedback, and monitoring of data flows
>>>    * Highly configurable along several dimensions of quality of service
>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>> high throughput, and priority based queuing
>>>    * Fine-grained data provenance for all data received, forked,
>>> joined, cloned, modified, sent, and ultimately dropped as data reaches
>>> its configured end-state
>>>    * Component-based extension model along well defined interfaces
>>> enabling rapid development and effective testing
>>> == Background ==
>>> Reliable and effective dataflow between systems can be difficult
>>> whether you're running scripts on a laptop or have a massive
>>> distributed computing system operated by numerous teams and
>>> organizations.  As the volume and rate of data grows and as the number
>>> of systems, protocols, and formats increase and evolve so too does the
>>> complexity and need for greater insight and agility.  These are the
>>> dataflow challenges that NiFi was built to tackle.
>>> NiFi is designed in a manner consistent with the core concepts
>>> described in flow-based programming as originally documented by J.
>>> Paul Morrison in the 1970s.  This model lends itself well to visual
>>> diagramming, concurrency, componentization, testing, and reuse.  In
>>> addition to staying close to the fundamentals of flow-based
>>> programming, NiFi provides integration system specific features such
>>> as: guaranteed delivery; back pressure; ability to gracefully handle
>>> backlogs and data surges; and an operator interface that enables
>>> on-the-fly data flow generation, modification, and observation.
>>> == Rationale ==
>>> NiFi provides a reliable, scalable, manageable and accountable
>>> platform for developers and technical staff to create and evolve
>>> powerful data flows.  Such a system is useful in many contexts
>>> including large-scale enterprise integration, interaction with cloud
>>> services and frameworks, business to business, intra-departmental, and
>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>> integrates with several others.  We also anticipate developing
>>> extensions for several other ASF projects such as Cassandra, Kafka,
>>> and Storm in the near future.
>>> == Initial Goals ==
>>>    * Ensure all dependencies are compliant with Apache License version
>>> 2.0 and all that all code and documentation artifacts have the correct
>>> Apache licensing markings and notice.
>>>    * Establish a formal release process and schedule, allowing for
>>> dependable release cycles in a manner consistent with the Apache
>>> development process.
>>>    * Determine and establish a mechanism, possibly including a
>>> sub-project construct, that allows for extensions to the core
>>> application to occur at a pace that differs from the core application
>>> itself.
>>> == Current Status ==
>>> === Meritocracy ===
>>> An integration platform is only as good as its ability to integrate
>>> systems in a reliable, timely, and repeatable manner.  The same can be
>>> said of its ability to attract talent and a variety of perspectives as
>>> integration systems by their nature are always evolving.  We will
>>> actively seek help and encourage promotion of influence in the project
>>> through meritocracy.
>>> === Community ===
>>> Over the past several years, NiFi has developed a strong community of
>>> both developers and operators within the U.S. government.  We look
>>> forward to helping grow this to a broader base of industries.
>>> === Core Developers ===
>>> The initial core developers are employed by the National Security
>>> Agency and defense contractors.  We will work to grow the community
>>> among a more diverse set of developers and industries.
>>> === Alignment ===
>>>  From its inception, NiFi was developed with an open source philosophy
>>> in mind and with the hopes of eventually being truly open sourced.
>>> The Apache way is consistent with the approach we have taken to date.
>>> The ASF clearly provides a mature and effective environment for
>>> successful development as is evident across the spectrum of well-known
>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>> dependencies with several more ASF projects, including Accumulo, Avro,
>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>> This existing alignment with Apache and the desired community makes
>>> the Apache Incubator a good fit for NiFi.
>>> == Known Risks ==
>>> === Orphaned Products ===
>>> Risk of orphaning is minimal.  The project user and developer base is
>>> substantial, growing, and there is already extensive operational use
>>> of NiFi.
>>> === Inexperience with Open Source ===
>>> The initial committers to NiFi have limited experience with true open
>>> source software development.  However, despite the project origins
>>> being from closed source development we have modelled our behavior and
>>> community development on The Apache Way to the greatest extent
>>> possible.  This environment includes widely accessible source code
>>> repositories, published artifacts, ticket tracking, and extensive
>>> documentation. We also encourage contributions and frequent debate and
>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>> and in-person meet-ups.  We are committed to the ideals of open source
>>> software and will eagerly seek out mentors and sponsors who can help
>>> us quickly come up to speed.
>>> === Homogenous Developers ===
>>> The initial committers of NiFi come from a limited set of entities
>>> though we are committed to recruiting and developing additional
>>> committers from a broad spectrum of industries and backgrounds.
>>> === Reliance on Salaried Developers ===
>>> We expect NiFi development to continue on salaried time and through
>>> volunteer time.  The initial committers are paid by their employers to
>>> contribute to this project.  We are committed to developing and
>>> recruiting participation from developers both salaried and
>>> non-salaried.
>>> === Relationship with other Apache Projects ===
>>> As described in the alignment section, NiFi is already heavily
>>> dependent on other ASF projects and we anticipate further dependence
>>> and integration with new and emerging projects in the Apache family.
>>> === An Excessive Fascination with the Apache Brand ===
>>> We respect the laudable Apache brand and that is certainly a factor in
>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>> provide a great deal of energy and capability to the ASF through this
>>> project.  We will be sensitive to and respectful of any overuse of the
>>> Apache brand and ensure our focus remains on how we benefit the Apache
>>> community.
>>> === Documentation ===
>>> At this time there is no NiFi documentation on the web.  However, we
>>> have extensive documentation included within the application that
>>> details usage of the many functions.  We will be rapidly expanding the
>>> available documentation to cover things like installation, developer
>>> guide, frequently asked questions, best practices, and more.  This
>>> documentation will be posted to the NiFi wiki at
>>> == Initial Source ==
>>> NiFi has been in active development since late 2006 with contributions
>>> from dozens of developers and feedback from hundreds of users and
>>> developers.  The core codebase is written in Java and includes
>>> detailed Javadocs and feature documentation.
>>> == Source and Intellectual Property Submission ==
>>> Previously referred to as Niagarafiles, the NiFi code and
>>> documentation materials will be submitted by the National Security
>>> Agency.  NiFi has been developed by a mix of government employees and
>>> private companies under government contract.  Material developed by
>>> the government employees is in the public domain and no U.S. copyright
>>> exists in works of the federal government.  For the contractor
>>> developed material in the initial submission, the U.S. Government has
>>> sufficient authority to open source per DFARS 252.227-7014.  NSA has
>>> submitted the Software Grant Agreement and Corporate Contributor
>>> License Agreement to the Apache Software Foundation.
>>> == External Dependencies ==
>>> We have at least one dependency on an LGPL library which we will
>>> promptly address.  Otherwise, we believe all current dependencies are
>>> compatible with the ASF guidelines.  Our dependency licenses come from
>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>> Eclipse Public v1, MIT, CDDL v1.
>>> == Cryptography ==
>>> Consistent with we believe
>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>> cryptographic algorithms but is designed to use algorithms provided by
>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>> These cryptographic algorithm providers are used to support SSL,
>>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>>> In the event that it becomes necessary we will engage with appropriate
>>> Apache members to ensure we file any necessary paperwork or clarified
>>> any cryptographic export license concerns.
>>> == Required Resources ==
>>> === Mailing Lists ===
>>>    *
>>>    *
>>>    *
>>>    *
>>> === Source Control ===
>>> NiFi requests use of Git for source control
>>> (git://  We request a writeable Git repo for
>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>> INFRA ticket for this.
>>> === Issue Tracking ===
>>> JIRA NiFi (NIFI)
>>> === Initial Committers ===
>>>    * Brandon Devries<brandon.devries at gmail dot com>
>>>    * Matt Gilman<matt.c.gilman at gmail dot com>
>>>    * Tony Kurc<trkurc at gmail dot com>
>>>    * Mark Payne<markap14 at hotmail dot com>
>>>    * Adam Taft<adam at adamtaft dot com>
>>>    * Joseph Witt<joewitt at gmail dot com>
>>> === Affiliations ===
>>>    * Brandon Devries (Requitest, Inc.)
>>>    * Matt Gilman (Raytheon)
>>>    * Tony Kurc (National Security Agency)
>>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>>    * Adam Taft (Requitest, Inc.)
>>>    * Joseph Witt (National Security Agency)
>>> == Sponsors ==
>>> === Champion ===
>>>    * Benson Margulies (Basis Technology)<bimargulies at apache dot org>
>>> === Nominated Mentors ===
>>>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>    * Brock Noland (Cloudera)<brock at apache dot org>
>>>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>> === Sponsoring Entity ===
>>> We request the Apache Incubator to sponsor this project.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message