nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Gilman <matt.c.gil...@gmail.com>
Subject Re: NiFi: command-line interface ?
Date Fri, 25 Mar 2016 14:35:18 GMT
Dmitry,

/controller/templates

Will return all the templates that have been created/imported in your NiFi
instance. Created/Imported in this case does not mean instantiated in your
data flow on the canvas. It's just the definition of the template that has
been defined. Each template will have an identifier, name, description, etc.

/controller/process-groups/{process-group-id}/template-instance

Will instantiate a given template (that's previously been created/imported)
into the process group specified as the path parameter. The process group
id can be the identifier for the desired group or the alias 'root' if the
target is the root level process group. Additionally, this endpoint will
require an x and y coordinate. This is the origin (upper left) of the
bounding box of all components in the template. This will drive the
placement in the graph on your canvas.

You can start and stop individual components or perform bulk actions using
the encapsulating process group. For instance, you can start all components
in a given process group.

I don't know what 'done', 'in progress', or 'failed with error' mean in the
context of your data flow. You'll need to monitor the status of the various
processor that drive those states in your workflow.

Everything that you see in the UI is happening through the REST API. I
would suggest using NiFi with your browser's Developer Tools open and
seeing the requests that are being made. You should be able to extract your
desired curl command from there. I'm pretty sure that Chrome even supports
a 'Copy as cUrl' option.

Matt

On Fri, Mar 25, 2016 at 9:27 AM, Dmitry Goldenberg <dgoldenberg@hexastax.com
> wrote:

> Matt,
>
> >> The "official" API is the REST API (see link above), so you can use
> "curl" or something like that to interact with the NiFi instance.
>
> I'm looking at the REST API documentation
> <https://nifi.apache.org/docs/nifi-docs/rest-api/>. What I'd like to be
> able to do is: look up a template, instantiate it as a dataflow, kick off
> the dataflow, and get its status as "done", "failed with error", or "in
> progress".  How much of this can be done via the REST API?
>
> It looks like template instantiation can be done as
> /controller/process-groups/{process-group-id}/template-instance.  I've got
> questions about this as, firstly I don't have any process groups defined in
> my template, so must I organize the template as a process group? The
> request parameters ask for a templateId; is that just the name of the
> template?
>
> On the question of kicking off the dataflow. Basically I see that
> Get/ingress processors (at least typically) already take care of their
> scheduling strategies. However, for a DevOps case, devops engineers will
> likely want to be in control of scheduling of dataflows and monitoring
> their status, not via the GUI but through a command-line interface (which,
> as we've discussed, could be done with something like curl via the REST
> API).
>
> Would these abilities to schedule (or kick off once, immediately) a given
> dataflow outside of the GUI and monitor its status be enhancement requests
> then for NiFi?  I'd imagine this would affect the existing Ingress
> processors, as they'd need an alternative to the timer-driven and
> CRON-driven scheduling strategies, to allow for ad hoc invocations and for
> invocations made through the REST API.
>
> Thanks,
> - Dmitry
>
>
>
>
> ---------- Forwarded message ----------
> From: Matt Burgess <mattyb149@gmail.com>
> Date: Sun, Mar 20, 2016 at 3:23 PM
> Subject: Re: NiFi: command-line interface ?
> To: users@nifi.apache.org
>
>
> Dmitry,
>
> With regards to nifi-client (I am the author), that exception occurs when
> the flow has been changed externally and the shell has not recognized it.
> What the result of the following command?
>
> nifi.currentVersion
>
> If it is -1, then I recommend restarting the shell. It should be a
> non-negative integer, a -1 usually indicates there was an error
> communicating with the NiFi instance, or it couldn't parse the response.
> The response containing the current revision of the NiFi workflow can be
> obtained from your browser or "curl" using the following REST endpoint:
>
> /nifi-api/controller/revision
>
> It should be a JSON document, but there isn't any code in nifi-client
> currently to force a JSON response. I will add that shortly for version 0.4
> (which will allow you to add processors, interact with process groups,
> etc.)  What version of nifi-client and NiFi are you using?
>
> nifi-client is a Groovy object model for NiFi concepts (processor, e.g.),
> and performing operations on the objects results in calls to the NiFi REST
> API (https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) To
> answer the questions:
>
> 1) Your nifi-client example code should work as-is (but see above)
>
> 2) There is at least one other solution for command-line deployment:
> https://github.com/aperepel/nifi-api-deploy.  It uses Groovy scripts and
> YAML configuration files to interact with the REST API.  These are
> contributions from the community and not "official" command-line tools for
> Apache NiFi. Having said that, I'm quite sure we'd both appreciate any
> suggestions (and especially contributions!) to make these better.
>
> The "official" API is the REST API (see link above), so you can use "curl"
> or something like that to interact with the NiFi instance. However Andrew
> and I have both used Groovy to abstract the complexities of the API away
> from the user.
>
> Regards,
> Matt
>
> On Sun, Mar 20, 2016 at 2:37 PM, Dmitry Goldenberg <
> dgoldenberg123@gmail.com> wrote:
>
>> I see some posts on the Web regarding NiFi's command-line interface.  One
>> of them lead me to a client library
>> <https://github.com/mattyb149/nifi-client> that one can use via a Groovy
>> shell.
>>
>> I seem to have been able to instantiate my sample dataflow through this
>> interface but when I tried kicking off the dataflow by starting its first
>> processor, I got an exception "processor:[id:6c71c1c9-1220-44a2-a020-8bd3c82d1692,
>> state:RUNNING]] groovyx.net.http.HttpResponseException: Conflict.
>>
>> In the canvas, I've made sure nothing is running but I'm getting this
>> exception (full stack trace below).
>>
>> My questions are:
>>
>> a) is this the right way to kick off a given existing dataflow;
>> b) are there any other command-line options for performing operations on
>> dataflows (and templates)?
>>
>> The CLI options are often critical from the DevOps perspective, mostly
>> for being able to script things up.
>>
>> Thanks.
>> - Dmitry
>>
>> ---------------------------------------
>>
>>
>> *nifi-shell:*000*>* nifi.processors.'GetXmlFiles'.start()
>>
>> [revision:[version:713],
>> processor:[id:6c71c1c9-1220-44a2-a020-8bd3c82d1692, state:RUNNING]]
>>
>> groovyx.net.http.HttpResponseException: Conflict
>>
>> at
>> groovyx.net.http.HTTPBuilder.defaultFailureHandler(HTTPBuilder.java:651)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at
>> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>>
>> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>>
>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1210)
>>
>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1077)
>>
>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
>>
>> at groovy.lang.Closure.call(Closure.java:426)
>>
>> at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:503)
>>
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)
>>
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)
>>
>> at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
>>
>> at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:434)
>>
>> at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:366)
>>
>> at groovyx.net.http.HTTPBuilder$request.call(Unknown Source)
>>
>> at nifi.client.Processor.updateState(Processor.groovy:149)
>>
>> at nifi.client.Processor$updateState$0.callCurrent(Unknown Source)
>>
>> at nifi.client.Processor.start(Processor.groovy:127)
>>
>> at nifi.client.Processor$start.call(Unknown Source)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
>>
>> at groovysh_evaluate.run(groovysh_evaluate:3)
>>
>> at groovysh_evaluate$run.call(Unknown Source)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>
>> at groovysh_evaluate$run.call(Unknown Source)
>>
>> at
>> org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
>>
>> at org.codehaus.groovy.tools.shell.Evaluator$evaluate.call(Unknown Source)
>>
>> at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:187)
>>
>> at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
>>
>> at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
>>
>> at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
>>
>> at
>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at
>> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>>
>> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>>
>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1210)
>>
>> at
>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>>
>> at
>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
>>
>> at
>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:158)
>>
>> at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
>>
>> at
>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at
>> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>>
>> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>>
>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1210)
>>
>> at
>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>>
>> at
>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
>>
>> at
>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
>>
>> at java_lang_Runnable$run.call(Unknown Source)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
>>
>> at org.codehaus.groovy.tools.shell.Groovysh.run(Groovysh.groovy:587)
>>
>> at org.codehaus.groovy.tools.shell.Groovysh$run$0.callCurrent(Unknown
>> Source)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
>>
>> at org.codehaus.groovy.tools.shell.Groovysh.run(Groovysh.groovy:552)
>>
>> at org.codehaus.groovy.tools.shell.Groovysh$run.call(Unknown Source)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
>>
>> at org.codehaus.groovy.tools.shell.Main.startGroovysh(Main.groovy:186)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>
>> at
>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
>>
>> at org.codehaus.groovy.tools.shell.Main.main(Main.groovy:152)
>>
>>
>>
>
>
>

Mime
View raw message