nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John McGinn <figgie...@sbcglobal.net>
Subject Re: Compiling custom processor
Date Wed, 17 Oct 2018 18:52:45 GMT
Thanks Bryan.

That was a mistake on my part. I thought that I could use part of the NiFi standard processors
in my custom processor. I suppose this also means it's not possible to extend a standard processor
class, say Wait, and override a method, for instance, onTrigger(), to do different work. Instead,
as you state, I should copy the portions I would need to my custom processor. Is that accurate?

In the end, as I was looking through how to modify Wait, I think it is bigger than the need
that I have, so I restarted with a new generated processor project, and just added the cache
service, and appropriate relationships, and simple logic to control flow through my processor.
I'll work on getting a github account and uploading my simple processor.

Thanks for the assistance
John McGinn

--------------------------------------------
On Tue, 10/16/18, Bryan Bende <bbende@gmail.com> wrote:

 Subject: Re: Compiling custom processor
 To: figgie123@sbcglobal.net
 Date: Tuesday, October 16, 2018, 4:45 PM
 
 You shouldn't really try to
 depend on code inside other NARs like
 standard processors, you'll want to copy
 WaitNotifyProtocol into your
 own NAR.
 On
 Tue, Oct 16, 2018 at 4:40 PM John McGinn <figgie123@sbcglobal.net>
 wrote:
 >
 > The
 commons-lang3-3.7.jar file is located in the lib/bootstrap
 directory, therefore, I figured it could be considered
 provided. I've since change it to normal scope, and it
 is included in my NAR and that error goes away. So, this
 > then leans on the next error I get, which
 is no class found for the WaitNotifyProtocol in
 nifi.processors.standard. Is there an easy way to figure out
 what NiFi class/object I'm using, and what NAR should be
 listed as provided? I tried to add
 nifi-standard-processors-nar, but that doesn't exist for
 maven, nor does it exist in the lib directory. I looked
 inside the nifi-standard-nar NAR file, and the
 nifi-standard-processors is there, and inside that is
 WaitNotifyProtocol, but yet, my maven won't build if I
 have nifi-standard-nar listed, either with type of NAR, or
 scope of provided.
 >
 --------------------------------------------
 > On Tue, 10/16/18, Bryan Bende <bbende@gmail.com>
 wrote:
 >
 >  Subject:
 Re: Compiling custom processor
 >  To: dev@nifi.apache.org,
 figgie123@sbcglobal.net
 >  Date: Tuesday, October 16, 2018, 4:01
 PM
 >
 >  Any reason
 why the commons-lang3
 >  dependency is
 provided and not normal
 >  compile
 scope?
 >
 >  Generally
 the only cases where a dependency
 > 
 should be provided are
 >  when a parent
 NAR
 >  provides it (the API scenario
 earlier), or when it
 >  is a jar
 directly in lib which means it is on
 > 
 the classpath of
 >  everything (like
 nifi-api
 >  or slf4j).
 >  On Tue, Oct 16, 2018 at 3:47 PM John
 McGinn
 >  <figgie123@sbcglobal.net>
 >  wrote:
 >  >
 >  > I have a
 > 
 directory /home/user/nifi-nars, which is listed in the
 >  nifi.properties file located under the
 nifi-assembly
 >  directory from the
 nifi-master code. That's where I
 > 
 redropped my NAR file at.
 >  >
 >  > The error was "due to
 uncaught
 >  Exception:
 java.lang.NoClassDefFoundError:
 > 
 org/apache/commons/lang3/StringUtils". I added a
 >  provided dependency of version 3.4 for
 commons-lang3 in my
 >  processors
 pom.xml. That worked initially, but with the
 >  redropping of the NAR file, the error
 occurs. The stacktrace
 >  points to the
 session.get() lambda function trying to use
 >  StringUtils.isBlank(). (Again, this is a
 copy and renamed
 >  class of the Wait
 Processor. Trying to turn it into a Yield,
 >  as it were, hold unless it's free to
 move forward.)
 >  >
 > 
 --------------------------------------------
 >  > On Tue, 10/16/18, Bryan Bende
 <bbende@gmail.com>
 >  wrote:
 >  >
 >  >  Subject:
 > 
 Re: Compiling custom processor
 >  > 
 To: dev@nifi.apache.org
 >  >  Date: Tuesday, October 16, 2018,
 3:34
 >  PM
 > 
 >
 >  >  There
 >  shouldn't be any issue
 >  >  doing
 > 
 that, generally you just copy the
 > 
 >
 >  new
 > 
 >  NAR into the lib directory and
 > 
 restart.
 >  >
 > 
 >  What
 >  were the specifics of the
 class not found
 >  >  error?
 >  >
 >  >  On
 Tue, Oct 16, 2018 at
 >  >  3:24 PM
 John McGinn <figgie123@sbcglobal.net>
 >  >  wrote:
 > 
 >  >
 >  >  > And
 I'm
 >  >
 > 
 back.
 >  >  >
 >  >
 >  > So, I
 had my
 >  >  processor running,
 >  no errors, so things were good.
 >  >
 >  >
 >  >  > I then add 2 if
 >  >  statements with logger messages,
 run the
 >  same mvn command,
 >  >  copied the new NAR
 >  on top of the old one, same versionId
 and
 >  >  all, and now I get a class
 not found
 >  error, that didn't
 >  >  occur before I
 >  stopped, redeployed and started. And
 I
 >  >  can't delete the event
 in the queue
 >  even though source
 >  >  and destination
 >  are stopped.
 > 
 >  >
 >  >  > Is there
 problems when redeploying
 >  a
 >  >  custom processor NAR on top of
 >  itself? Even if the NiFi
 >  >  instance is
 >  stopped? Serialization issues?
 >  >
 >  >
 >  >  > Frustratedly
 >  >  yours,
 > 
 >  > John
 >  McGinn
 >  >  >
 > 
 >
 >  >
 > 
 >
 > 
 --------------------------------------------
 >  >  > On Tue, 10/16/18, John
 McGinn
 >  <figgie123@sbcglobal.net>
 >  >  wrote:
 > 
 >  >
 >  >  >  Subject:
 >  >
 >  Re:
 Compiling custom processor
 >  > 
 >
 >  To: dev@nifi.apache.org
 >  >  >  Date: Tuesday, October 16,
 2018,
 >  1:21
 > 
 >  PM
 >  >
 > 
 >
 >  >  >  Thanks
 >  >  Bryan,
 > 
 >  >
 >  >  >  I
 >  >
 >  restarted
 with a fresh maven
 >  >  >
 >  >  generate, and only added the API
 to the
 >  processor pom,
 >  >  and
 >  > 
 >  then the API-NAR to the NAR
 > 
 pom.
 >  >  I then figured out a
 >  >  >  subsequent
 >  >
 >  dependency
 (lang3), and got that in, and now my
 > 
 >  >  processor shows up in local
 >  NiFi
 >  > 
 instance.
 >  >
 > 
 >
 >  >  >
 >  >
 >  Regarding
 the copying of the standard
 >  >
 >  >
 >  > 
 processor, when I copied Wait
 >  over, I
 renamed it from
 >  >  Wait
 >  >  >  to MyProcessorWait to
 be
 >  unique,
 > 
 >  and not copied. I'm
 > 
 just
 >  >  >
 >  >
 >  attempting
 changes to Wait without modifying the actual
 >  >  one.
 > 
 >  >  Also,
 >  thanks on the
 >  >  META-INF/services file
 >  information.
 > 
 >  >
 >  >  >  Now I can
 debug and test.
 >  >  >
 >  >  >
 > 
 Sincerely,
 >  >  >  John
 McGinn
 >  >  >
 >  >  >
 > 
 >
 > 
 --------------------------------------------
 >  >  >  On Tue, 10/16/18, Bryan
 Bende
 >  <bbende@gmail.com>
 >  >  >  wrote:
 >  >
 >  >
 >  >  >   Subject: Re: Compiling
 >  custom
 >  > 
 >  processor
 >  >  >   To:
 dev@nifi.apache.org
 >  >  >   Date: Tuesday, October
 16,
 >  2018,
 > 
 >  9:52
 >  >
 >  >  AM
 >  > 
 >
 >  >
 > 
 >   In general, if your processor
 > 
 >
 >  >   uses a controller
 service then the
 >  >  >  
 processors pom file needs a
 > 
 provided
 >  >  >  dependency
 >  >  >   on
 > 
 >  the
 >  API of the CS,
 >  >  >   and your NAR
 >  pom
 >  > 
 needs
 >  >
 > 
 >   a NAR dependency on the NAR
 > 
 >
 >  where the
 > 
 >  >  CS API
 >  >  >  
 is.
 >  >
 > 
 >
 >  >  >   Example is
 >  >  >
 > 
 >  shown here
 >  in the section
 linking
 >  >  >
 >  >  processors and
 >  >
 >  >  
 controller
 >  >  >   services
 -
 >  https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 >  >  >
 > 
 >  >
 >  Also, you do not
 >  >  want to
 > 
 >  >   include NiFI's standard
 >  >  processors in
 >  >
 >  > 
 your
 >  >  >   own NAR, this
 >  will result in two
 >  >  >  copies
 >  of
 >  >  >  
 every
 >  >  standard processor.
 >  >  >   You will want
 >  >  to
 >  > 
 >   copy
 >  whatever code you need
 into
 >  >  your
 >  >  >  own NAR.
 >  >
 >  >
 >  >  >   In you last statement,
 >  if you
 >  > 
 >   made a new processor
 >  in
 >  >  >  
 nifi-standard-processors
 >  and it
 >  >  didn't
 > 
 >  >  show
 >  >
 >  >   up, it is most likely
 >  >  >
 > 
 because you didn't
 >  >  >  
 update
 >  META-INF/services file to
 >  >  >
 > 
 include the new
 >  >  >
 >  >  processor.
 > 
 >  >
 >   On Tue, Oct 16, 2018
 >  >  at
 >  > 
 >   9:36 AM John McGinn
 >  > 
 >   <amruginn-nifi@yahoo.com.invalid>
 >  >  >   wrote:
 >  >
 >  >  
 >
 >  >  >   > Ok, I am
 >  >  >
 > 
 >  far from a
 >  maven expert, and
 am
 >  >  >
 > 
 >  struggling on this
 >  > 
 >   problem.
 >  >
 >  >   >
 > 
 >  >   > I
 >  > 
 created
 >  >  >   a
 >  new project using the
 >  >  maven
 > 
 generate
 >  >  >  process,
 and
 >  >  compiled
 >  >  >
 >  that
 sample processor and
 >  >
 >  everything
 >  > 
 >  was fine, and I
 >  could
 >  >  >   see it in my local
 NiFi
 >  instance. I
 >  >  >  then copied over
 >  the
 >  >  > 
  Wait processor on top of
 >  the
 >  >  >  MyProcessor.java class,
 >  changed
 >  > 
 >   the package name, and
 >  the
 class
 >  >  name,
 >  >  >  and attempted to
 >  >  >   compile. Got errors due
 to
 >  >  dependency
 >  >  >
 > 
 issues. This is where
 >  >  >  
 I get
 >  confused.
 >  >  >
 > 
 >
 >  >
 > 
 >  >   > In
 >  >  >
 >  >  the
 > 
 processor directory, I modify the
 > 
 >
 >  >
 > 
 >  pom.xml with a
 >  >  >  
 dependency line for
 >  > 
 nifi-api,
 >  >  >
 >  nifi-utils,
 > 
 >  >
 >  >  >
 >  >
 > 
 nifi-distribute-cache-client-service-api and
 >  >  >  
 nifi-standard-processors.
 >  This
 >  >  seemed
 > 
 >
 >  >  to make things work,
 >  >  >
 >  and I
 get a 33 meg NAR file, which
 >  >
 >  >  contains lots of JARs. I
 >  >
 >  >   load
 that up, and I get a problem
 >  > 
 with
 >  >  >
 >  CalculateStats, or
 >  >  >   similar,
 >  because of
 >  > 
 >
 >  RecordReaderFactory. I go back
 in and
 >  >
 > 
 >   include
 >  >  >
 >  > 
 nifi-record-serialization-services, and
 >  it compiles,
 > 
 >  >   and the JAR is
 > 
 included in the NAR
 >  >  > 
 file now,
 >  and I still get
 >  >  >   class not
 >  found.
 >  > 
 >
 >  >
 > 
 >
 >  >  >   >
 >  >  >
 > 
 >  I've
 >  also tried to change
 the
 >  >  >
 >  >  dependencies to a status
 >  >  >   of
 > 
 >
 >  provided, and therefore get a
 >  >  >
 > 
 minimal
 >  >  25k or so NAR file
 >  >  >   with no JARs
 >  >  included. I get the same
 >  >  >  issue with
 >  >
 >  the class
 >  >  >   not found.
 >  >  >   >
 >  >
 >  >   >
 I
 >  >  had
 > 
 >  >   also attempted to copy
 > 
 Wait.java
 >  >  to
 >  >
 >  > 
 AnotherWait.java within
 >  > 
 >
 >   the actually NiFi source code,
 and
 >  >
 >  it
 >  >  >  compiles, and shows up
 >  >  >   in the JARs and NARs
 as
 >  I'd
 > 
 >  expect,
 >  >  >  but I
 cannot get to
 >  >  >   it
 within my NiFi instance.
 >  > 
 >   >
 >  >
 >  >   > Can
 > 
 >  someone let me know
 >  what
 I'm
 >  >  >
 >  >  doing
 > 
 >  >
 >  wrong.
 >  >  >
 > 
 >
 >  >
 > 
 >  >   > Thanks,
 >  > 
 >   > John McGinn
 >  > 
 >

Mime
View raw message