Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E1D32200BCB for ; Thu, 10 Nov 2016 01:53:47 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E02FB160AFD; Thu, 10 Nov 2016 00:53:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D53CF160AFA for ; Thu, 10 Nov 2016 01:53:46 +0100 (CET) Received: (qmail 46948 invoked by uid 500); 10 Nov 2016 00:53:41 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 46936 invoked by uid 99); 10 Nov 2016 00:53:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2016 00:53:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4A2BB18396C for ; Thu, 10 Nov 2016 00:53:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2 X-Spam-Level: ** X-Spam-Status: No, score=2 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nH1teqtijKbB for ; Thu, 10 Nov 2016 00:53:36 +0000 (UTC) Received: from mail-pf0-f182.google.com (mail-pf0-f182.google.com [209.85.192.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 93C365F23C for ; Thu, 10 Nov 2016 00:53:35 +0000 (UTC) Received: by mail-pf0-f182.google.com with SMTP id 189so135687128pfz.3 for ; Wed, 09 Nov 2016 16:53:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :message-id:references:to; bh=2eMx9fHujNbItNTSKEmty8JXdNPJ36B+wxIsyuAaj9Q=; b=IBntdgdQyAFwBcRqr+FzOk9/QtUpWCuvDGLt2gWkDEGixP2wZhLRb8XBeFDNce7EEM nkCZjf9DD05avXIPrD4tqIgJB1c/q0sc28D33LD22AycdbcIkSZOdR7aSgi6+GeBebqS lpglFZ2VtKwZngUDoQmxjkZgm8/XEUTUCGOvpcgwKqWY8jwb15nnxdFvj6ArU/rbJ3Rl eFkXJPhQL7aD79d6Ag1yUOt10JKgNHhmUz9S+snS+WR2IejkOwLn5QOVPr26l/DgO0eX QWlxR+UUSYDaWGfdJS1bWZLA5cnyXiacO0ZRvIhwLSmUD1knDCI/Qv0ILbQIgsZ7PTtP lgeg== X-Gm-Message-State: ABUngvckb5l7/T7rX0qX5xvLW7QNnUcIcqT0rMHKnO4BxkU0UU+jX+Zw0bMFRtZGWrJi6w== X-Received: by 10.98.214.20 with SMTP id r20mr4425506pfg.59.1478739214010; Wed, 09 Nov 2016 16:53:34 -0800 (PST) Received: from [192.168.1.7] (cpe-172-91-141-241.socal.res.rr.com. [172.91.141.241]) by smtp.gmail.com with ESMTPSA id 123sm1967100pfe.41.2016.11.09.16.53.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 09 Nov 2016 16:53:33 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_F0223645-FD99-4295-A2CB-596768B91C78"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Getting the number of logs X-Pgp-Agent: GPGMail From: Andy LoPresto In-Reply-To: <17844F08-9B5A-49B0-93C3-D50B2A032A1C@capitalone.com> Date: Wed, 9 Nov 2016 16:54:41 -0800 Message-Id: <70A296DB-0E4F-4B14-B668-1CBD1ECA288D@apache.org> References: <17844F08-9B5A-49B0-93C3-D50B2A032A1C@capitalone.com> To: dev@nifi.apache.org X-Mailer: Apple Mail (2.3124) archived-at: Thu, 10 Nov 2016 00:53:48 -0000 --Apple-Mail=_F0223645-FD99-4295-A2CB-596768B91C78 Content-Type: multipart/alternative; boundary="Apple-Mail=_7BDD2597-1C5E-4E2B-90AC-74287690FD27" --Apple-Mail=_7BDD2597-1C5E-4E2B-90AC-74287690FD27 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Sai, I=E2=80=99d suggest you look at using a ControllerStatusReportingTask = [1], which monitors the processor and provides statistics from that = component. If you need to use this data within NiFi, you can also use = SiteToSiteProvenanceReportingTask [2], which can export provenance = events as data that can be consumed by (the same or a different) = instance of NiFi. Both of these may be overkill for your use case (the = provenance reporting task will offload all of the provenance events from = the application), and if so, you may be able to use counters [3] to do = this quickly and easily (but be aware that the values are just held in = memory, so if you=E2=80=99re writing to ES hourly, you should be ok, but = they won=E2=80=99t persist across restart). Your initial thought to use = ExecuteScript would also work. I believe Joe Percivall had done some work on SEP and window/aggregate = calculations before. That may also help with what you are doing. Joe Percivall=C2=B74:51 PM Here's the link to the processor: = https://github.com/JPercivall/nifi/blob/newRollingState/nifi-nar-bundles/n= ifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi= /processors/standard/RollingWindowAggregator.java Here's the ticket: = https://issues.apache.org/jira/browse/NIFI-1682?jql=3Dproject%20%3D%20NIFI= %20AND%20text%20~%20%22rolling%20window%22 Keep in mind that this work is old and will need to be updated. I do = have a pending PR for UpdateAttribute with State though: = https://github.com/apache/nifi/pull/319 [1] = https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controll= er.ControllerStatusReportingTask/index.html = [2] = https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reportin= g.SiteToSiteProvenanceReportingTask/index.html = [3] = https://community.hortonworks.com/questions/50622/apache-nifi-what-are-cou= nters-in-nifi.html = Andy LoPresto alopresto@apache.org alopresto.apache@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 9, 2016, at 8:14 AM, Peddy, Sai = wrote: >=20 > Hi All, >=20 > I=E2=80=99m currently working on a use case to be able to track the = number of individual logs that come in and put that information in = ElasticSearch. I wanted to see if there is an easy way to do this and = whether anyone had any good ideas? >=20 > Current approach I am considering: Route the Log Files coming in =E2=80=93= to a Split Text & Route Text Processor to make sure no empty logs get = through and get the individual log count when files contain multiple = logs =E2=80=93 At the end of this the total number of logs are visible = in the UI queue, where it displays the queueCount, but this information = is not readily available to any processor. Current thought process is = that I can use the ExecuteScript Processor and update a local file to = keep track and insert the document into elastic search hourly. >=20 > Any advice would be appreciated >=20 > Thanks, > Sai Peddy >=20 > ________________________________________________________ >=20 > The information contained in this e-mail is confidential and/or = proprietary to Capital One and/or its affiliates and may only be used = solely in performance of work or services for Capital One. The = information transmitted herewith is intended only for use by the = individual or entity to which it is addressed. If the reader of this = message is not the intended recipient, you are hereby notified that any = review, retransmission, dissemination, distribution, copying or other = use of, or taking of any action in reliance upon this information is = strictly prohibited. If you have received this communication in error, = please contact the sender and delete the material from your computer. --Apple-Mail=_7BDD2597-1C5E-4E2B-90AC-74287690FD27 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Sai,

I=E2=80= =99d suggest you look at using a ControllerStatusReportingTask [1], = which monitors the processor and provides statistics from that = component. If you need to use this data within NiFi, you can also use = SiteToSiteProvenanceReportingTask [2], which can export provenance = events as data that can be consumed by (the same or a different) = instance of NiFi. Both of these may be overkill for your use case (the = provenance reporting task will offload all of the provenance events from = the application), and if so, you may be able to use counters [3] to do = this quickly and easily (but be aware that the values are just held in = memory, so if you=E2=80=99re writing to ES hourly, you should be ok, but = they won=E2=80=99t persist across restart). Your initial thought to use = ExecuteScript would also work. 

I believe Joe Percivall had done some = work on SEP and window/aggregate calculations before. That may also help = with what you are doing. 

Joe Percivall=C2=B74:= 51 PM
Here's the link to the = processor: https://github.com/JPercivall/nifi/blob/newRollingState/nifi-na= r-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/= apache/nifi/processors/standard/RollingWindowAggregator.javaHere's the ticket: https://issues.apache.org/jira/browse/NIFI-1682?jql=3Dproject%2= 0%3D%20NIFI%20AND%20text%20~%20%22rolling%20window%22

Keep in mind that this work is old and will need to be = updated. I do have a pending PR for UpdateAttribute with State though: = https://github.com/apache/nifi/pull/319


[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.ni= fi.controller.ControllerStatusReportingTask/index.html
[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.ni= fi.reporting.SiteToSiteProvenanceReportingTask/index.html
[3] https://community.hortonworks.com/questions/50622/apache-nifi-w= hat-are-counters-in-nifi.html

Andy = LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC = B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 9, 2016, at 8:14 AM, Peddy, Sai <Sai.Peddy@capitalone.com> wrote:

Hi = All,

I=E2=80=99m currently working on a use = case to be able to track the number of individual logs that come in and = put that information in ElasticSearch. I wanted to see if there is an = easy way to do this and whether anyone had any good ideas?

Current approach I am considering: Route the = Log Files coming in =E2=80=93 to a Split Text & Route Text Processor = to make sure no empty logs get through and get the individual log count = when files contain multiple logs =E2=80=93 At the end of this the total = number of logs are visible in the UI queue, where it displays the = queueCount, but this information is not readily available to any = processor. Current thought process is that I can use the ExecuteScript = Processor and update a local file to keep track and insert the document = into elastic search hourly.

Any advice = would be appreciated

Thanks,
Sai Peddy

________________________________________________________

The information contained in this e-mail is = confidential and/or proprietary to Capital One and/or its affiliates and = may only be used solely in performance of work or services for Capital = One. The information transmitted herewith is intended only for use by = the individual or entity to which it is addressed. If the reader of this = message is not the intended recipient, you are hereby notified that any = review, retransmission, dissemination, distribution, copying or other = use of, or taking of any action in reliance upon this information is = strictly prohibited. If you have received this communication in error, = please contact the sender and delete the material from your computer.

= --Apple-Mail=_7BDD2597-1C5E-4E2B-90AC-74287690FD27-- --Apple-Mail=_F0223645-FD99-4295-A2CB-596768B91C78 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJYI8VSAAoJEDxu9lsvfe9plmwP/jtU0YVfOLIMFjZgXsr5JQ7l JM0UycIJ85w6PJ5sceE1PkybcSZG2yE1pRgDfSRFf0tjmiJDDm41vVhnGXjtFqEb Ci3URRI5GT64xWzHFyp6NItYCCtlH25gmiEn9JIPYcDoQEjdcugz9mJDfg7W30vU YwyBw7SC4roB+ZQVpUBM9T0PYS8FKn0Gq2yG9eNY25vb/4s46PZbMGqsh2emVQ5U IWyRCJ9FCImXQaJRaurvdrKGpvJJTsMmoe91N6w8jRFrxtci4gFRtjZ6R4gbc8B8 zff0Ru6i6V9vK0Qd2uEHIti2/389ciZvaKuW2ovOyj325nDZIrYMrQOZ/f207vFa 1EUUm5nIvb0viNHq54VBqZCUm+JWHN5yC7wT/Ix/qO++23fGCtKJv14MbGMBQbaU 89pW6+/mwStzqbkhOtNrBykJHGOA55xJwPeT2mVxNQedXKw/+JUaiGHBw/bDfHNr +Q6PXFrq40otedjT+ANOL0EEoUVGRxffR1O+UcBf+7Tgexgplza1+e4rkUsLUSTx qC6YQVWr0UpSi9I7D08W/6ZOEIsVknPocyqwFKG1DMf4uJxJC2fZfAIWKI7CV10b vV8dUN+ncrZBH9HZLoRbWK0p8xJ+Ps4aHkJlX5G9E4aK3Dfutp3WQ3hqUmZrQLyM vzApZxCDrWMBcfGVjq8J =eYh6 -----END PGP SIGNATURE----- --Apple-Mail=_F0223645-FD99-4295-A2CB-596768B91C78--