nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Percivall <joeperciv...@yahoo.com.INVALID>
Subject Re: Getting the number of logs
Date Thu, 10 Nov 2016 16:00:18 GMT
I hadn't received Andy's response before I sent mine to the User's list. For tracking, here
is a link to my response: http://mail-archives.apache.org/mod_mbox/nifi-users/201611.mbox/raw/%3C215272108.1093060.1478740617664@mail.yahoo.com%3E/2
Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joepercivall@yahoo.com
 

    On Wednesday, November 9, 2016 7:53 PM, Andy LoPresto <alopresto@apache.org> wrote:
 

 Sai,
I’d suggest you look at using a ControllerStatusReportingTask [1], which monitors the processor
and provides statistics from that component. If you need to use this data within NiFi, you
can also use SiteToSiteProvenanceReportingTask [2], which can export provenance events as
data that can be consumed by (the same or a different) instance of NiFi. Both of these may
be overkill for your use case (the provenance reporting task will offload all of the provenance
events from the application), and if so, you may be able to use counters [3] to do this quickly
and easily (but be aware that the values are just held in memory, so if you’re writing to
ES hourly, you should be ok, but they won’t persist across restart). Your initial thought
to use ExecuteScript would also work. 
I believe Joe Percivall had done some work on SEP and window/aggregate calculations before.
That may also help with what you are doing. 
Joe Percivall·4:51 PM
Here's the link to the processor: https://github.com/JPercivall/nifi/blob/newRollingState/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RollingWindowAggregator.java
Here's the ticket: https://issues.apache.org/jira/browse/NIFI-1682?jql=project%20%3D%20NIFI%20AND%20text%20~%20%22rolling%20window%22

Keep in mind that this work is old and will need to be updated. I do have a pending PR for
UpdateAttribute with State though: https://github.com/apache/nifi/pull/319

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html[3] https://community.hortonworks.com/questions/50622/apache-nifi-what-are-counters-in-nifi.html
Andy LoPrestoalopresto@apache.orgalopresto.apache@gmail.comPGP Fingerprint: 70EC B3E5 98A6
5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 9, 2016, at 8:14 AM, Peddy, Sai <Sai.Peddy@capitalone.com> wrote:
Hi All,

I’m currently working on a use case to be able to track the number of individual logs that
come in and put that information in ElasticSearch. I wanted to see if there is an easy way
to do this and whether anyone had any good ideas?

Current approach I am considering: Route the Log Files coming in – to a Split Text &
Route Text Processor to make sure no empty logs get through and get the individual log count
when files contain multiple logs – At the end of this the total number of logs are visible
in the UI queue, where it displays the queueCount, but this information is not readily available
to any processor. Current thought process is that I can use the ExecuteScript Processor and
update a local file to keep track and insert the document into elastic search hourly.

Any advice would be appreciated

Thanks,
Sai Peddy

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.




   
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message