nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject Re: [DISCUSS] NiFi 1.1.0 release
Date Tue, 22 Nov 2016 10:09:56 GMT
I'm biased, but I think the new Query / Scroll Elasticsearch processors are
pretty big ;)

On Tue, Nov 22, 2016 at 9:02 AM, Andy LoPresto <alopresto@apache.org> wrote:

> Important to call out in the security section that
> login-identity-providers.xml is now covered by encrypted config, and
> flow.xml.gz processor property encryption/nifi.sensitive.props.key value
> can now be migrated by the command-line tool (second part pending
> completion of NIFI-3024).
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 21, 2016, at 9:30 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
> matt
>
> i'll add a wiki page or set of instructions linked from the release notes.
>
> all,
>
> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
> highlights.  The items noted are as follows.  Will likely reduce this
> down further for the release notes but wanted to put this out in case
> folks have things they think are really important to highlight.
>
> - Core Improvements:
>   - Performance: Session Migration
>   - Stability: Cluster Management
>   - Developer: Framework supports easy user driven classloader extension
>   - Expression Language: Now supports base64 and hex encoded values
> and Math functions
>   - Repositories now support rollback
>   - Faster startup due to more efficient state restoration algorithm
> - UX Improvements:
>   - Visual Backpressure Indicator
>   - Introduced more colors to better highlight actions and components
>   - Performance: Validate non-running components
>   - Provenance graph image can be exported
>   - Cron Scheduling for Primary node tasks now supported
> - Updated versions
>   - Azure Event Hub 0.9.0
>   - Spark 2.0.1
>   - Hadoop 2.7.x
> - New/Improved Processors
>   - new Fetch/Put Elastic Search 5.0
>   - new ParseCEF to parse CEF formatted logs
>   - improve ExtractEmail now supports TNEF files
>   - new Validate CSV
>   - improved Solr processors now support SSL and Kerberos
>   - new Websocket client and server processors
> - New Utility
>   - Zookeeper Migrator (move from one zookeeper to another)
> - Security
>   - Restricted Processors
>   - Site-to-site now supports port forwarding
>   - Improved Policy Management UX
> - Migration Notes:
>   - Restricted Processors
>   - Twitter Processor Removed
>
> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess <mattyb149@apache.org>
> wrote:
>
> Is there a good spot for us to put instructions on how to build the
> Twitter processor and/or the Social Media NAR in the meantime? Maybe a
> Wiki page or something simple to say "go to this directory, run this
> Maven command, drop the NAR into your deployment..." ?
>
> On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
> Team,
>
> We appear to be very close.  Andy is working NIFI-3024 but otherwise
> it is focus on testing.
>
> I'm going to prep the RC and release notes now.  Unfortunately the
> twitter changes for json.org will need to remain.  Consensus forming
> on the legal-discuss thread regarding a grace period has been elusive
> and we're already prepared to make the right steps so we'll just need
> to take that on by being empathetic to the user base.
>
> Thanks
> Joe
>
> On Thu, Nov 17, 2016 at 7:37 AM, Andre <andre-lists@fucs.org> wrote:
>
> Andy,
>
> Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0
> is set to include a number of security related improvements.
>
>
>
> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto <alopresto@apache.org>
> wrote:
>
> Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been
> added to my plate for this release. Coordinated with Joe Witt and they
> should both be included.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3050
> [2] https://issues.apache.org/jira/browse/NIFI-3051
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 16, 2016, at 12:08 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
> Team
>
> There is a thread on apache legal-discuss that might allow for a
> graceperiod of continued usage of the json library.  Am going to keep
> a close eye on this and if VP Legal approves we'll be able to keep the
> twitter processors in which is definitely a good thing.  Will advise
>
> Thanks
> Joe
>
> On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende <bbende@gmail.com> wrote:
>
> I've noticed an issue with the per-instance class loading capability
> introduced in NIFI-2909 where the additional classpath resources can get
> incorrectly removed from the class loader.
>
> I was able to reproduced this with a unit test and have a fix ready. I
> believe this is important and needs to go in for the 1.1 release, going to
> re-open NIFI-2909 and submit a PR shortly.
>
> -Bryan
>
> On Wed, Nov 16, 2016 at 8:11 AM, Matt Gilman <matt.c.gilman@gmail.com>
> wrote:
>
> I have two items that I would like to wrap up prior to creating an RC for
> 1.1.0. NIFI-2949 addresses some UX issues around Remote Process Group port
> configuration. The work is already completed and I will be reviewing it
> this today. Additionally, following recent interest on the mailing list,
> I'd like to knock out NIFI-3020. This will allow an admin to configure a
> strategy for user identity when logging in via LDAP. Specifically, it will
> support usage of the DN (the default and current implementation) as well as
> the username the user logged in as. I should be able to have a PR up for
> this work later today.
>
> Thanks!
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-2949
> [2] https://issues.apache.org/jira/browse/NIFI-3020
>
>
> On Tue, Nov 15, 2016 at 8:00 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
> The code is within the twitter4j library itself.  I filed a request to
> twitter4jg.  The most likely case is we will need to submit a PR to them.
> However, I don't see this as something that should delay the release.  We
> can provide instructions for folks wanting to use the processor during
>
> the
>
> time we cannot make it available in a convenient manner.  I will provide
>
> a
>
> meaningful comment about this in release notes and pointers on what folks
> can do in the meantime.
>
> On Nov 15, 2016 7:41 PM, "Andy LoPresto" <alopresto@apache.org> wrote:
>
> I understand there was a discussion thread within the NiFi community
>
> for
>
> this as well and I missed responding to that at that time. It just
>
> seems
>
> to
>
> me like JSON processing is necessary for GetTwitter, which is
>
> incredibly
>
> useful for demonstrating NiFi’s ability to read from a high volume
>
> stream
>
> out of the box. With NIFI-3019 (Remove GetTwitter from default build),
>
> is
>
> there any related effort to substitute an acceptable replacement JSON
> library to restore this functionality?
>
> [1] https://issues.apache.org/jira/browse/NIFI-3019
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 15, 2016, at 4:36 PM, Andy LoPresto <alopresto@apache.org>
>
> wrote:
>
>
> I’m working with Bryan Rosander to close out NIFI-3024, NIFI-2655, and
> NIFI-2653. I believe Matt Burgess is working on NIFI-3011 and we
> investigated some alternate TLS config options for the new version of
>
> the
>
> client library.
>
> Is there any alternative to excluding the GetTwitter processor? Using
> Johnzon [1] or the Android re-implementation [2] discussed in the
>
> mailing
>
> list thread?
>
> [1] https://johnzon.apache.org/
> [2] https://developer.android.com/reference/org/json/package-
>
> summary.html
>
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 15, 2016, at 3:58 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
> Team
>
> Very happy to see that we are down to three items remaining tagged to
> 1.1.0.  Solid effort over the recent weeks to close the gap including
>
> work
>
> to get past the now category x Jason dependency we had.  The most
>
> notable
>
> impact from that is the wildly popular GetTwitter processor, the fav
>
> new
>
> nifi user and demo processor, can no longer be included in the default
> build.  It is optionally available if users choose to build and use it
>
> but
>
> we won't distribute binaries that have it.
>
> I see some review movement on some patch available but untagged items.
>
> I plan to kick off the 1.1.0 rc work soon. Perhaps Thurs or Fri. Anyone
> have any outstanding items?
>
> Thanks
> Joe
>
> On Nov 8, 2016 2:12 PM, "Joe Witt" <joe.witt@gmail.com> wrote:
>
> Ryan
>
> Not officially but I think we should try to close this thing out and
> start a vote in the next week or two at most.
>
> I'm going through the tickets again now.  There is also a new issue of
> the json-p license falling out of favor in Apache legal terms and
> becoming Category-X.  Am looking into that now.
>
> Thanks
> Joe
>
> On Tue, Nov 8, 2016 at 2:05 PM, Ryan Ward <ryan.ward2@gmail.com>
>
> wrote:
>
>
> Joe - Is there a target date for 1.1?
>
> On Thu, Oct 27, 2016 at 10:50 AM, Joe Witt <joe.witt@gmail.com> wrote:
>
> Team,
>
> Just an update on things with working toward an Apache NiFi 1.1.0
> release.  There are still about 33 JIRAs there now and some are
> awaiting review and are some are under active progress. Yet there is
> good traction and progress. I think we should just stay vigilant with
> what makes it in and keep working it down.  So let's please shoot for
> a couple weeks from now.  If it is ready sooner I'll jump on it.
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 9:06 AM, Joe Witt <joe.witt@gmail.com> wrote:
>
> Team,
>
> There are 31 open JIRAs at present tagged to Apache NiFi 1.1.0.  Let's
> avoiding putting more in there for now at least without a discussion.
> Of the 31 JIRAs there the vast majority need review so we should be
> able to close these down fairly quickly as long as we don't let the
> list grow.
>
> Thanks
> joe
>
> On Fri, Oct 14, 2016 at 4:39 PM, Edgardo Vega <edgardo.vega@gmail.com>
>
> wrote:
>
> Joe,
>
> Appreciate the offer it isn't my PR. I was just using it as an
>
> example.
>
> All
>
> mine are currently closed, which I greatly appreciate.
>
> Cheers,
>
> Edgardo
>
> On Friday, October 14, 2016, Joe Witt <joe.witt@gmail.com> wrote:
>
> Edgardo,
>
> You mentioned a PR from August. I'd be happy to help you work that
> through review.
>
> Thanks
> Joe
>
> On Fri, Oct 14, 2016 at 10:45 AM, Edgardo Vega <
>
> edgardo.vega@gmail.com
>
> <javascript:;>> wrote:
>
> I have agreed that at this point a release is important. My goal
>
> was
>
> try
>
> to
>
> squeeze in a much goodness as possible into the release, but the
>
> important
>
> bug fixes should come first. Getting 1.x into a state where the
>
> release
>
> notes don't say that it is geared toward developers and testers is
>
> really
>
> huge.
>
> I think Nifi is a great community otherwise I would participate in
>
> the
>
> mailing list, create Jira tickets and pull requests. I am only
>
> trying to
>
> strengthen the great thing that is going on here. We can always do
>
> better.
>
> I was not trying to put down this community only to participate and
>
> make
>
> it
>
> better. I think this conversation is an indication of how great
>
> this
>
> community is.
>
> Maybe I am being sensitive about this issue and trying to
>
> strengthen
>
> the
>
> nifi community even more, after coming from a conference where it
>
> was
>
> reported there was lots of excitement at first and now the
>
> participation
>
> in
>
> the community has really died down and they are struggling. I don't
>
> want
>
> to
>
> see that happen here.
>
> Cheers,
>
> Edgardo
>
>
>
>
> On Fri, Oct 14, 2016 at 9:37 AM, Andre <andre-lists@fucs.org
>
> <javascript:;>> wrote:
>
>
> Edgardo,
>
> Thank you for your feedback. We hear your comments and as a
>
> committer I
>
> can
>
> share we are constantly looking to improve the PR process, having
>
> already
>
> taken many of the steps you suggest.
>
> However, it is important to notice that the number of PRs should
>
> not be
>
> seen as a metric of engagement by the development community: Most
>
> of us
>
> will submit PRs so that our work can be carefully reviewed by our
>
> peers
>
> and
>
> some of us will use JIRA patches to provide contributions.
>
> Having said that, it is true that some PRs may sit idle for a long
>
> time
>
> and
>
> we are working to improve this pipeline.
>
> It was therefore no coincidence that I  browsed most of the PRs
>
> performing
>
> a triage of items that have been superseded or diverged from the
>
> current
>
> code base.
>
> In fact, less than a month ago the dev team closed a number of
>
> stalled
>
> and
>
> superseded PRs (commit cc5e827aa1dfe2f376e9836380ba63c15269eea8).
>
> Despite all the above, I think Joe has a point. The master
>
> contain a
>
> series
>
> of important bug fixes and suspect the community would benefit
>
> from
>
> a
>
> release sooner rather than later.
>
> Once again, thank you for your feedback and contribution. It is
>
> good to
>
> have you here.
>
> Andre
>
> On Fri, Oct 14, 2016 at 11:30 PM, Edgardo Vega <
>
> edgardo.vega@gmail.com
>
> <javascript:;>>
>
> wrote:
>
> Joe - You are correct I was mentioning the PRs that are
>
> currently
>
> open.
>
>
> Regardless of how it happens reducing the count of open PRs I
>
> believe
>
> to
>
> be
>
> extremely important. Maybe I was hoping that the release could
>
> be
>
> a
>
> forcing
>
> function to make that happen. I believe that developers are more
>
> willing
>
> to
>
> contribute when they see that their PRs will actually be able
>
> accepted
>
> and
>
> merged into the code base. Having a low number of open PRs in
>
> progress
>
> is a
>
> great indication that the main nifi developers are fully engaged
>
> with
>
> the
>
> community.
>
> There are a few PRs that don't have any comments from committers
>
> at
>
> all.
>
> I
>
> found one from August in that state. If that was my PR I don't
>
> think I
>
> would be so willing to put another one in anytime soon. I do get
>
> that
>
> sometime PRs get stalled by the originator, if so maybe a rule
>
> about
>
> closing them after a certain amount of time or being taken over
>
> by a
>
> core
>
> contributor if they think it worthwhile.
>
> I would like to shoutout to James Wing on my last PR he was
>
> quick
>
> to
>
> review, provided great comments, testing, and even some
>
> additional
>
> code.
>
> It
>
> was a great PR experience.
>
> Cheers,
>
> Edgardo
>
>
>
> On Thu, Oct 13, 2016 at 4:14 PM, Joe Percivall <
>
> joepercivall@yahoo.com <javascript:;>.
>
> invalid> wrote:
>
> Joe, I think you misread. Edgardo is referring to the Pull
>
> Requests
>
> that
>
> are currently open, not the tickets assigned to the 1.1.0
>
> version.
>
>
> I think these goals (releasing 1.1.0 and cutting down the PR
>
> count)
>
> should
>
> be two different efforts. Doing a thorough job reviewing
>
> takes a
>
> significant amount of time from both the reviewer and
>
> contributor.
>
> In
>
> order
>
> to cut it down significantly would take much longer than a
>
> couple
>
> days.
>
>
> Also there has already been a lot of great new features and
>
> bug
>
> fixes
>
> contributed to the 1.X line and I don't think it's worth
>
> holding up
>
> a
>
> 1.1.0
>
> release for tickets not assigned to this fix version. As an
>
> added
>
> bonus
>
> though, I think many of the tickets tagged as 1.1.0 have PRs
>
> already
>
> open
>
> so closing those will make a large dent in the PR count.
>
>
> Joe
>
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joepercivall@yahoo.com <javascript:;>
>
>
>
> On Thursday, October 13, 2016 3:58 PM, Joe Witt <
>
> joe.witt@gmail.com
>
> <javascript:;>>
>
> wrote:
>
>
>
> There are less than 30 right now.  Many of the roughly 90+
>
> JIRAs
>
> opened on 1.1.0 were easily dispositioned to 1.2.0 or closed
>
> or
>
> just
>
> had fix versions removed.
>
> We will need to have a push over the next bunch of days to
>
> deal
>
> with
>
> reviewing/merging/moving the remaining items.
>
> Thanks
> Joe
>
>
> On Thu, Oct 13, 2016 at 3:49 PM, Edgardo Vega <
>
> edgardo.vega@gmail.com <javascript:;>>
>
> wrote:
>
> Joe,
>
> There are 75 PRs currently open. Why not make a push over
>
> the
>
> next
>
> bunch
>
> of
>
> days to get them closed and then cut the release after that.
>
> Cheers,
>
> Edgardo
>
> On Thu, Oct 13, 2016 at 12:44 PM, Joe Witt <
>
> joe.witt@gmail.com
>
> <javascript:;>>
>
> wrote:
>
>
> Team,
>
> There have been a ton of bugs fixed a few nice features.  I
>
> would
>
> like
>
> to move to get Apache NiFi 1.1.0 release going pretty much
>
> based
>
> on
>
> where we are now and plan to move most tickets to a new
>
> Apache
>
> NiFi
>
> 1.2.0 version.  We can try to get back on our roughly 6-8
>
> week
>
> release
>
> schedule and shoot for a mid to late Nov release for NiFi
>
> 1.2.0
>
> this
>
> way as well. Please advise if anyone has any other views on
>
> this. In
>
> the mean time I'll get the wheels in motion so you'll be
>
> seeing a
>
> lot
>
> of JIRA/issue updates to move version around.
>
> Thanks
> Joe
>
> On Thu, Oct 13, 2016 at 12:02 PM, Tony Kurc <
>
> trkurc@gmail.com
>
> <javascript:;>>
>
> wrote:
>
> Sounds good Joe. I have no issue to you doing the rm'ing
>
> for
>
> it.
>
>
> On Oct 13, 2016 8:19 AM, "Joe Witt" <joe.witt@gmail.com
>
> <javascript:;>> wrote:
>
>
> Team,
>
> There are a lot of great fixes and improvements on the
>
> master
>
> line
>
> now
>
> and we're at a good time window to start pushing for a
>
> release.
>
> There
>
> are, however, about 90+ JIRAs assigned to 1.1.0 which
>
> are
>
> open.
>
> I'm
>
> going to go through them and remove fix versions where
>
> appropriate.
>
>
> I'm happy to take on RM task for this release though if
>
> someone
>
> else
>
> would like to take that on please advise.
>
> Thanks
> Joe
>
>
>
>
>
> --
> Cheers,
>
> Edgardo
>
>
>
>
>
> --
> Cheers,
>
> Edgardo
>
>
>
>
>
> --
> Cheers,
>
> Edgardo
>
>
>
>
> --
> Cheers,
>
> Edgardo
>
> Sent from Gmail Mobile
>
>
>
>
>
>
>
>
>
>
>


-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message