nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Gilman <matt.c.gil...@gmail.com>
Subject Re: Nifi connection queue monitoring.
Date Tue, 03 Nov 2015 18:54:15 GMT
Chakri,

The start and end times are not recorded in those statistics. Those
statistics are the value for what's happing right now or within the last
five minutes.

You would have to check provenance for events originating from that
processor and then look at the duration of each event to know the stop time.
When you query provenance, you need to create your provenance request and
then continue polling it until it has completed. Since the provenance
request (espicially when clustered) can be long running we've broke it out
into multiple requests. The gist is basically..

POST https://{host}:{port}/nifi-api/controller/provenance
GET https://
{host}:{port}/nifi-api/controller/provenance/{provenance-request-id}
... (as many GETs as necessary to complete) ...
DELETE https://
{host}:{port}/nifi-api/controller/provenance/{provenance-request-id}

Again, check out the REST API docs and the developer tools in your browser
to see the requests in action and see how the search criteria is specified.

Another option would be to create a Reporting Task (another NiFi extension
point) and access the provenance events that way programmatically in Java.

Thanks.

Matt

On Tue, Nov 3, 2015 at 1:26 PM, Chakrader Dewaragatla <
Chakrader.Dewaragatla@lifelock.com> wrote:

> Matt – How do I get the start time of a processor ?
>  we have a getSftp processor with time driven schedule (6h interval). I
> would like to monitor the start and finish time for that interval, where do
> I see the captured stats ?
>
> Thanks,
> -Chakri
>
> From: Chakrader Dewaragatla <chakrader.dewaragatla@lifelock.com>
> Date: Monday, November 2, 2015 at 4:12 PM
>
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: Nifi connection queue monitoring.
>
> Thanks Matt, this should help us find the stats.
>
> Thanks,
> -Chakri
>
> From: Matt Gilman <matt.c.gilman@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Monday, November 2, 2015 at 4:39 PM
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: Nifi connection queue monitoring.
>
> Chakri,
>
> The configuration and the status are decoupled from one another so we
> don't need to pull back the entire configuration when a user wants to
> refresh the statistics frequently. Currently, the only endpoint available
> for retrieving status is based on a Process Group. That end point is
>
>
> https://{host}:{port}/nifi-api/controller/process-groups/{process-group-id}/status?recursive={true|false}
>
> If you want to get the status of every component, you can use the alias
> "root" for the top most process group id and set recursive to TRUE. If you
> want a specific connection, you can use the process group id of the Process
> Group that connection resides in. The best way to see these requests in
> action is to open up your Developer Tools in your browser and watch the
> requests the UI makes.
>
> Let me know if you have any more questions. Thanks!
>
> Matt
>
> On Mon, Nov 2, 2015 at 6:11 PM, Chakrader Dewaragatla <
> Chakrader.Dewaragatla@lifelock.com> wrote:
>
>>  Hi,
>> Does nifi has any REST api to monitor the queue size for a connection ?
>>  I am trying to query a connection to see if it has any data
>> queued/backlogged. I will use the backlog queue size to determine a
>> workflow is chocking with massive incoming data or one of the downstream
>> processor stopped processing.
>>
>>   Following api show the connection status, not queue size.
>>
>>   /controller/process-groups/{process-group-id}/connections/{id}
>>
>>   Thanks,
>>   -Chakri
>> ------------------------------
>> The information contained in this transmission may contain privileged and
>> confidential information. It is intended only for the use of the person(s)
>> named above. If you are not the intended recipient, you are hereby notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all copies
>> of the original message.
>> ------------------------------
>>
>
> ------------------------------
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>

Mime
View raw message