nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Bateman <russell.bate...@perfectsearchcorp.com>
Subject Re: A few NiFi ReST questions...
Date Sat, 09 Jul 2016 15:47:54 GMT
Andy,

Thanks. Yes, I know how Ajax works having used it before. It's looking at
it happening in the browser developer tools that I've never learned to do.
I've been using </> RESTED so that I can set Accept application/json which
works perfectly.

Russ


On Fri, Jul 8, 2016 at 10:55 PM, Andy LoPresto <alopresto@apache.org> wrote:

> Russell,
>
> No worries. The Javascript application that runs the UI is simply invoking
> Ajax (Asynchronous Javascript and XML) requests to normal HTTP endpoints
> provided by the embedded Jetty server in NiFi. You don’t need to understand
> anything about front-end development to monitor these requests using the
> Dev Tools. In fact, you could simply type the entire API endpoint into the
> browser address bar and receive the XML (or JSON-formatted, if you provide
> a header requesting it) response from the server directly. If you are more
> comfortable on the command-line or in an IDE, you can use Java, Groovy,
> etc. or even curl to invoke the API purely through code. There are also API
> tools (no endorsement, but some are listed here [1]) for hitting these APIs
> from a GUI application.
>
> Here is a good starting point for using the Network tab of the developer
> tools window [2].
>
> Glad to hear you have been able to write some custom processors. Let us
> know if you have any additional questions.
>
> [1]
> https://stackoverflow.com/questions/4486658/what-tools-do-you-use-to-test-your-public-rest-api
> [2] http://commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art034
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 7:42 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> (Make that "not much of a web page developer". I'm a back-end Java guy
> mostly, don't much value my UI skills and so have stayed away from the
> browser except for straight HTML and CSS.)
>
> On Fri, Jul 8, 2016 at 8:39 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> Thanks, Andy and Matt, I had actually tried out the Developer Tools
> earlier today as suggested by the doc, but, I'm not much of a web
> developer, don't have a lot of familiarity with those tools in Chrome or in
> Firefox, and I couldn't see anything going on (in Firefox) that was
> helpful. I may have to grow strong that way.
>
> For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
> appearance, we can move up, but you point out something we're already
> beginning to pay attention to: incompatiblities. Still we're committed to
> (and love!) the technology. I've written a dozen custom processors (for
> proprietary ETL stuff) already and my colleague has done the same.
>
> Thanks, again.
>
>
> On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <alopresto@apache.org>
> wrote:
>
> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example”  on the canvas,
> I get this response:
>
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> <processGroupStatusEntity>
> <revision>
> <clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
> </revision>
> <processGroupStatus>
> <activeThreadCount>0</activeThreadCount>
> <id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
> <input>0 / 0 bytes</input>
> <name>NiFi Flow</name>
> <output>0 / 0 bytes</output>
> <processorStatus>
> <activeThreadCount>0</activeThreadCount>
> <groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
> <id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
> <input>0 / 0 bytes</input>
> <name>*Token Example*</name>
> <output>0 / 0 bytes</output>
> <read>0 bytes</read>
> <runStatus>Invalid</runStatus>
> <tasks>0</tasks>
> <tasksDuration>00:00:00.000</tasksDuration>
> <type>*Base64EncodeContent*</type>
> <written>0 bytes</written>
> </processorStatus>
> <queued>0 / 0 bytes</queued>
> <queuedCount>0</queuedCount>
> <queuedSize>0 bytes</queuedSize>
> <read>0 bytes</read>
> <received>0 / 0 bytes</received>
> <sent>0 / 0 bytes</sent>
> <statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
> <transferred>0 / 0 bytes</transferred>
> <written>0 bytes</written>
> </processGroupStatus>
> </processGroupStatusEntity>
>
> [1] https://github.com/aperepel/nifi-api-deploy
> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <matt.c.gilman@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my
> computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi
> instance is the root level process group. If you don't know the actual ID,
> you can use the alias 'root' as the path parameter for the process group
> ID.
>
> Please check out your browser's Dev Tools to see these API's in action.
> The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0
> release to align with the authorization policies of each resource. So the
> answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe,
> think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
> 1. How many times a processor processed anything?
> 2. How many flowfiles were processed?
> 3. How many flowfiles were produced?
> 4. Can I get this information per processor?
> 5. How can I tell different instances of the same processor apart?
> 6. Why can't I see the name I gave a processor in configuration? (For
> example, I named an instance of GetFile to "Get PDF files from test
> fodder".
> 7. How can I get a list of items on the NiFi canvas like processors and
> relationships.
> 8. How many times processors or a processor failed?
> 9. What is the profile of resource usage, like memory in use?
> 10. What is the profile of processor latency?
>    - flowfiles backed up awaiting processing
>    - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message