nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: Getting the number of logs
Date Thu, 10 Nov 2016 00:54:41 GMT
Sai,

I’d suggest you look at using a ControllerStatusReportingTask [1], which monitors the processor
and provides statistics from that component. If you need to use this data within NiFi, you
can also use SiteToSiteProvenanceReportingTask [2], which can export provenance events as
data that can be consumed by (the same or a different) instance of NiFi. Both of these may
be overkill for your use case (the provenance reporting task will offload all of the provenance
events from the application), and if so, you may be able to use counters [3] to do this quickly
and easily (but be aware that the values are just held in memory, so if you’re writing to
ES hourly, you should be ok, but they won’t persist across restart). Your initial thought
to use ExecuteScript would also work.

I believe Joe Percivall had done some work on SEP and window/aggregate calculations before.
That may also help with what you are doing.

Joe Percivall·4:51 PM
Here's the link to the processor: https://github.com/JPercivall/nifi/blob/newRollingState/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RollingWindowAggregator.java
Here's the ticket: https://issues.apache.org/jira/browse/NIFI-1682?jql=project%20%3D%20NIFI%20AND%20text%20~%20%22rolling%20window%22

Keep in mind that this work is old and will need to be updated. I do have a pending PR for
UpdateAttribute with State though: https://github.com/apache/nifi/pull/319


[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html>
[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html>
[3] https://community.hortonworks.com/questions/50622/apache-nifi-what-are-counters-in-nifi.html
<https://community.hortonworks.com/questions/50622/apache-nifi-what-are-counters-in-nifi.html>

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 9, 2016, at 8:14 AM, Peddy, Sai <Sai.Peddy@capitalone.com> wrote:
> 
> Hi All,
> 
> I’m currently working on a use case to be able to track the number of individual logs
that come in and put that information in ElasticSearch. I wanted to see if there is an easy
way to do this and whether anyone had any good ideas?
> 
> Current approach I am considering: Route the Log Files coming in – to a Split Text
& Route Text Processor to make sure no empty logs get through and get the individual log
count when files contain multiple logs – At the end of this the total number of logs are
visible in the UI queue, where it displays the queueCount, but this information is not readily
available to any processor. Current thought process is that I can use the ExecuteScript Processor
and update a local file to keep track and insert the document into elastic search hourly.
> 
> Any advice would be appreciated
> 
> Thanks,
> Sai Peddy
> 
> ________________________________________________________
> 
> The information contained in this e-mail is confidential and/or proprietary to Capital
One and/or its affiliates and may only be used solely in performance of work or services for
Capital One. The information transmitted herewith is intended only for use by the individual
or entity to which it is addressed. If the reader of this message is not the intended recipient,
you are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.


Mime
View raw message