nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Grande <apere...@gmail.com>
Subject Re: Lowering the barrier of entry
Date Mon, 28 Jan 2019 16:20:39 GMT
+1 it's so obvious it should've always been there :)

As for offline access. Well, every time I showed an offline help section to
every user thry were startled :) tells about awareness. The first hunch is
always to google insread of going to a locally hosted doc, so it won't be
an issue, IMO.

Andrew

On Mon, Jan 28, 2019, 8:13 AM Bryan Bende <bbende@gmail.com> wrote:

> What does everyone think about bumping the "Developer" section of the
> docs ahead of "Processors" so that it's easier to find?
>
> Here is what it would look like -
> https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
>
> I also added links to the "Contributor Guide" and the "Maven Projects"
> page since I think it would be helpful to make the Developer section
> be the one-stop place people look for development help, although I can
> see an argument for not mixing wiki content with the docs content.
>
> One issue would be if we want the docs to be fully usable in an
> off-line environment, then linking to the wiki won't work, so we could
> remove those links, or convert those pages to be part of the docs now
> that they are more stable.
>
> For reference, we already have some links in the docs to the wiki:
>
>
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
>
> On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <joe.witt@gmail.com> wrote:
> >
> > ...we can.  but the discussion is about how to both lower the bar and
> offer
> > more routes to the bar.
> >
> > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <ottobackwards@gmail.com
> wrote:
> >
> > > Why wouldn’t we make the archetypes do this?
> > >
> > >
> > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopresto@apache.org)
> > > wrote:
> > >
> > > James,
> > >
> > > I’m wondering if a page outlining a toy processor (something like
> > > `CountText` or `ReverseContent`) and doing a line-by-line annotation
> from a
> > > developer’s perspective would be helpful. It could be a few sections:
> > >
> > > 1. How to get to this point
> > > * running the maven archetype
> > > * choosing the directory to install to
> > > * putting the class name in the manifest file
> > > * etc.
> > > 2. The code
> > > * here’s the class
> > > * here’s what extending AbstractProcessor gets you, etc. A lot of this
> is
> > > currently in the Developer Guide, but in textbook mode
> > > * here’s how to modify some code (i.e. write one line of Java that
> switches
> > > it from straight content pass-through to reversing the text)
> > > * here’s how to make a unit test (introduce the TestRunner framework,
> etc.)
> > > 3. Running, building, installing
> > > * Run your unit test from the IDE/maven
> > > * Build the NAR module
> > > * Install the NAR in NiFi lib/ or custom/
> > > * Restart NiFi
> > > * See the NAR loaded in the log
> > > * Deploy the component on the canvas
> > >
> > > I imagine this being written more conversationally/blog-like than most
> of
> > > our current reference documentation to be used as a split-screen
> > > walkthrough. Each section could certainly link to the existing detailed
> > > documentation for various topics, like the processor lifecycle, etc.
> > >
> > > Does this sounds like something that would have helped you?
> > >
> > > Andy LoPresto
> > > alopresto@apache.org
> > > alopresto.apache@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > james.srinivasan@gmail.com>
> > > wrote:
> > > >
> > > > 9) Oh, and the wiki is a little hard to navigate and the contents
> rather
> > > patchy
> > > >
> > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > > <james.srinivasan@gmail.com> wrote:
> > > >>
> > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > >> realise I could and possibly should submit PRs :)
> > > >>
> > > >> 1) I'm used to Java and Maven, so used the archetype. It worked
> fine,
> > > >> it would have been nice it if set up unit tests for me.
> > > >> 2) The User and Developer documentation is great and comprehensive.
> > > >> Finding the developer docs is a little painful (handful of items at
> > > >> the end of a scrolling list of 200+ processors)
> > > >> 3) The Developer docs could possibly do with a little more clarity
> on
> > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > >> over the docs, it looks pretty clear now
> > > >> 4) Some example code for common operations e.g. getting/setting
> > > >> attributes or reading/writing/modifying flowfile content would be
> > > >> great.
> > > >> 5) When using existing processors for inspiration, best practices
> > > >> weren't always clear e.g. some generated properties inside
> > > >> getSupportedPropertyDescriptors(), others generated a private static
> > > >> list on init and returned that. Such differences are inevitable in
a
> > > >> large project, but it would be nice to have something blessed to
> start
> > > >> from.
> > > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > > screen)
> > > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> > > >> the great blog posts by Matt Burgess and others were invaluable
> > > >> 8) In case this all sounds too negative, NiFi is fab!
> > > >>
> > > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <aperepel@gmail.com>
> wrote:
> > > >>>
> > > >>> I am not against the archetype. But we need to spell out every
> step of
> > > the
> > > >>> way. I'd like to see a user thinking about their custom logic
ASAP
> > > rather
> > > >>> than fighting the tools to get started. Those steps should be
> > > brain-dead,
> > > >>> just reflexes, if you know what I mean. Hell, let them create
a
> custom
> > > >>> processor project or prototype in a script by accident even! :)
> > > >>>
> > > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbende@gmail.com>
> wrote:
> > > >>>
> > > >>>> That makes sense about the best practice for deploying to
an
> > > >>>> additional lib directory.
> > > >>>>
> > > >>>> So for the project structure you are saying it would be easier
to
> have
> > > >>>> a repo somewhere with essentially the same thing that is in
the
> > > >>>> archetype, but they just clone it and rename it themselves
(what
> the
> > > >>>> archetype does for you)?
> > > >>>>
> > > >>>> Something that I think would be awesome is if we could provide
a
> > > >>>> web-based project initializer that would essentially run the
> archetype
> > > >>>> behind the scenes and then let you download the archive of
the
> code,
> > > >>>> just like the spring-boot starter [1]. Not sure if their
> initializr is
> > > >>>> something that can be re-used and customized [2].
> > > >>>>
> > > >>>> The problem is we would need to host that somewhere.
> > > >>>>
> > > >>>> [1] https://start.spring.io/
> > > >>>> [2] https://github.com/spring-io/initializr
> > > >>>>
> > > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <
> aperepel@gmail.com>
> > > wrote:
> > > >>>>>
> > > >>>>> We assume they create new projects from archetypes every
day.
> They
> > > don't.
> > > >>>>>
> > > >>>>> We also assume they know how to deploy new NARs. Most
don't.
> > > Especially
> > > >>>> if
> > > >>>>> we want them to follow best practices and create an additional
> NAR
> > > >>>> bundles
> > > >>>>> directory entry im the config (vs dumping into nifi lib).
> > > >>>>>
> > > >>>>> I can attest that I feel a bit lost myself every time
I need to
> come
> > > back
> > > >>>>> to this and refresh my brain synapses. If we could make
these not
> > > require
> > > >>>>> any of that and make simple thinga dead simple....
> > > >>>>>
> > > >>>>> Andrew
> > > >>>>>
> > > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <bbende@gmail.com>
> wrote:
> > > >>>>>
> > > >>>>>> Andrew,
> > > >>>>>>
> > > >>>>>> I'm not disagreeing with your points, but I'm curious
how you
> see
> > > >>>>>> those two ideas being different from the processor
archetype
> and the
> > > >>>>>> wiki page with the archetype commands?
> > > >>>>>>
> > > >>>>>> Is it just that people don't know about it?
> > > >>>>>>
> > > >>>>>> -Bryan
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > >>>>>>
> > > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > >
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I think this ties into my other discuss thread
on refreshing
> the
> > > >>>>>> archetypes
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande
(
> aperepel@gmail.com
> > > )
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I consistently see my users struggling when they
move up the
> nifi
> > > >>>> food
> > > >>>>>>> chain and start looking at custom processors.
The good content
> > > about
> > > >>>>>>> prototyping processsors via scripting processors
and finalizing
> > > with
> > > >>>> a
> > > >>>>>> full
> > > >>>>>>> NAR bundle is everywhere but where it should be.
> > > >>>>>>>
> > > >>>>>>> A few simple changes could help (not *more* docs).
They are
> great,
> > > >>>> much
> > > >>>>>>> better than in many other projecta, but people
are already
> drowning
> > > >>>> in
> > > >>>>>>> those.
> > > >>>>>>>
> > > >>>>>>> How about:
> > > >>>>>>>
> > > >>>>>>> + ISP has a pre-populated processor sceleton.
A simple no-op to
> > > fill
> > > >>>> in
> > > >>>>>> is
> > > >>>>>>> miles better than a blank text area (which invokes
a blank
> stare).
> > > >>>>>>>
> > > >>>>>>> + As much as we may loook down on this, but...
A simple guide
> to a
> > > >>>> full
> > > >>>>>> NAR
> > > >>>>>>> build as a series of copy/paste commands.
> > > >>>>>>>
> > > >>>>>>> There's more, but this should fit the context
for now.
> > > >>>>>>>
> > > >>>>>>> Andrew
> > > >>>>>>>
> > > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <
> mikerthomsen@gmail.com
> > > >
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> One of the changes we should make is to create
a separate
> guide
> > > for
> > > >>>>>>> product
> > > >>>>>>>> vendors on how to build and maintain a bundle.
We're at that
> point
> > > >>>>>> where
> > > >>>>>>>> vendors will have to do it on their own as
extension
> providers, so
> > > >>>> it
> > > >>>>>>> would
> > > >>>>>>>> be very helpful for them to have a simple
and straight forward
> > > >>>> document
> > > >>>>>>>> showing them what should be there, best practices
for
> > > >>>> maintainability
> > > >>>>>> and
> > > >>>>>>>> where to announce it.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende
<bbende@gmail.com
> >
> > > >>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> I think we have a lot more documentation
than most projects,
> but
> > > >>>> I
> > > >>>>>>>>> think an issue is that content is scattered
in many different
> > > >>>>>>>>> locations, and some of the docs are huge
reference guides
> where
> > > >>>> it
> > > >>>>>> can
> > > >>>>>>>>> be hard to find all the pieces of what
you are trying to do.
> > > >>>>>>>>>
> > > >>>>>>>>> The first thing a new contributor wants
to do is get the code
> > > >>>> and run
> > > >>>>>>>>> a build, and we do have a quick-start
guide linked to on the
> > > >>>> site,
> > > >>>>>> but
> > > >>>>>>>>> I think there is a lot of extra information
in there that is
> not
> > > >>>>>>>>> really relevant to someone just wanting
get the code and
> build.
> > > >>>> We
> > > >>>>>>>>> could have separate guides per OS like
"Build NiFi on Linux",
> > > >>>> "Build
> > > >>>>>>>>> NiFi on Windows", etc, where each guide
was 4-5 steps like:
> > > >>>>>>>>>
> > > >>>>>>>>> - Clone repo
> > > >>>>>>>>> - checkout master
> > > >>>>>>>>> - run maven
> > > >>>>>>>>> - cd to assembly
> > > >>>>>>>>> - ./bin/nifi.sh
> > > >>>>>>>>>
> > > >>>>>>>>> The next thing they want to do is contribute
a change, and we
> > > >>>> have a
> > > >>>>>>>>> great contributor guide, but again I think
there could be a
> very
> > > >>>>>> short
> > > >>>>>>>>> tutorial for the most common steps:
> > > >>>>>>>>>
> > > >>>>>>>>> - fork repo
> > > >>>>>>>>> - clone fork
> > > >>>>>>>>> - create branch
> > > >>>>>>>>> - make changes
> > > >>>>>>>>> - push branch
> > > >>>>>>>>> - submit pr
> > > >>>>>>>>>
> > > >>>>>>>>> and then say something like "for a more
detailed description
> of
> > > >>>> the
> > > >>>>>>>>> contribution process, please reference
the Contributor
> Guide".
> > > >>>>>>>>>
> > > >>>>>>>>> If we then make these getting started
guides more prominent
> > > >>>> right in
> > > >>>>>>>>> the middle of the NiFi homepage, then
maybe they will be
> easier
> > > >>>> to
> > > >>>>>>>>> find for new community members.
> > > >>>>>>>>>
> > > >>>>>>>>> We can keep extending this idea to other
common tasks beyond
> just
> > > >>>>>>>>> building and contributing.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto
<
> > > >>>> alopresto@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi folks,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Based on some recent (and long-term)
experiences, I wanted
> to
> > > >>>>>> discuss
> > > >>>>>>>>> with the community what we could do to
lower the barrier of
> > > >>>> entry to
> > > >>>>>>>> using
> > > >>>>>>>>> & contributing to NiFi. I hope to
get some good feedback from
> > > >>>> both
> > > >>>>>>>>> long-time and newer members, and determine
some immediate
> > > >>>> concrete
> > > >>>>>>> steps
> > > >>>>>>>> we
> > > >>>>>>>>> can take.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Problems identified:
> > > >>>>>>>>>> * NiFi has a number of custom profiles,
so a simple “mvn
> clean
> > > >>>>>>> install”
> > > >>>>>>>>> in project root doesn’t get a new developer
up and running
> > > >>>>>> immediately
> > > >>>>>>>>>> * The API is very well defined, but
for new contributors, it
> > > >>>> can
> > > >>>>>> be a
> > > >>>>>>>>> challenge to know where to put functionality,
and building a
> > > >>>> custom
> > > >>>>>>>>> processor + NAR and deploying isn’t
a one-step process
> > > >>>>>>>>>> * Project size (and build size/time)
is large. This can
> > > >>>> restrict
> > > >>>>>> the
> > > >>>>>>>>> minimum hardware necessary, elongate the
development cycle,
> etc.
> > > >>>>>>>>>> * Some new users do not receive mailing
list replies
> > > >>>>>>>>>>
> > > >>>>>>>>>> Possible solutions:
> > > >>>>>>>>>> * On a clean git clone, “mvn clean
install” should build a
> > > >>>> working
> > > >>>>>>>>> instance. Maybe we provide a quickstart.sh
script to handle
> the
> > > >>>>>> default
> > > >>>>>>>>> maven build, change to the target directory,
and start NiFi?
> > > >>>>>>>>>> * Individual contributors have written
excellent blogs, and
> > > >>>>>>>>> documentation exists, but making it more
prominent or more
> easily
> > > >>>>>>>> accessed
> > > >>>>>>>>> could help?
> > > >>>>>>>>>> * Extension registry will solve all
the world’s problems
> > > >>>> (related
> > > >>>>>> to
> > > >>>>>>>>> bundling and build time)
> > > >>>>>>>>>> * Not sure about this one — I don’t
know if it’s because
> > > >>>> they’re
> > > >>>>>> not
> > > >>>>>>>>> subscribed, their mail client is blocking
them, etc.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I’ve said my bit, now I am eager
to hear from other
> community
> > > >>>>>> members
> > > >>>>>>>> on
> > > >>>>>>>>> their experiences, steps that helped them,
and suggestions
> for
> > > >>>> the
> > > >>>>>>> future
> > > >>>>>>>>> to continue to make the NiFi community
welcoming to new
> users.
> > > >>>>>> Thanks.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Andy LoPresto
> > > >>>>>>>>>> alopresto@apache.org
> > > >>>>>>>>>> alopresto.apache@gmail.com
> > > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F
D3C4 BACE 3C6E F65B
> 2F7D
> > > >>>> EF69
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message