Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
MIME-Version: 1.0
In-Reply-To: <4A3FFC38-12C7-43E5-9228-5678E3BE1F01@hpe.com>
References: <3A86C0AF-B702-4AD4-B3AE-6703291E93E8@hpe.com> <BLU436-SMTP21AB7087F88D4B13B141E2CEE30@phx.gbl>
 <CAGwx6xJPSAKyKB+SbK4JrpSCSxS0qmJSvTL_8woKKKSkN=a7pQ@mail.gmail.com>
 <CADnyFFJEWXFk4PFBm5S-m4Sxk2Sz80JJkv0JmrruiiqCumjStQ@mail.gmail.com>
 <CAEV8zdV+-N_UHH+9cZq1-Ue7xc_FKCr7Nud6hZkdi3GVE6230A@mail.gmail.com> <4A3FFC38-12C7-43E5-9228-5678E3BE1F01@hpe.com>
From: Joe Witt <joe.witt@gmail.com>
Date: Thu, 1 Sep 2016 13:28:25 +1000
Message-ID: <CALJK9a5jWfV_opx0WbKLqZqUuMY7FpE68LLbBFFinqPcwzrghw@mail.gmail.com>
Subject: Re: PostHTTP Penalize file on HTTP 5xx response
To: dev@nifi.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 01 Sep 2016 03:28:43 -0000

It will not be blocked by penalized things.  The queues are setup to
basically put those aside and move on to other things until their
penalty period passes. If you're seeing different behavior please
advise.

Thanks
Joe

On Thu, Sep 1, 2016 at 1:11 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) <chris.mcdermott@hpe.com> wrote:
> Thanks, everyone for the feedback. I=E2=80=99ll file a JIRA for this and =
see if I can find some time to address it.
>
> Does anyone have any thoughts on my related question?
>
> (with spelling and grammar corrections:)
>
> =E2=9E=A2 If a penalized file is routed to a funnel that=E2=80=99s s conn=
ect to a processor via a connection with the OldestFlowFileFirst  prioritiz=
er will the consumption of files from that connection be blocked until pena=
lization period is over?
>
>
>
> Chris McDermott
>
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
>
>
>
> On 8/31/16, 11:00 PM, "Matt Burgess" <mattyb149@apache.org> wrote:
>
>     Adam,
>
>     A PenalizeFlowFile processor could be pretty useful, please feel free
>     to file a New Feature Jira for this if you like.
>
>     In the meantime you could use ExecuteScript (with Groovy for this
>     example) and the following:
>
>     def flowFile =3D session.get()
>     if(!flowFile) return
>     flowFile =3D session.penalize(flowFile)
>     session.transfer(flowFile, REL_SUCCESS)
>
>     In this case the "success" relationship is awkward, it means you
>     successfully penalized the flow file. But then you can route it
>     back/forward to the appropriate processor. If you create a template
>     from this single processor, then dragging the template onto the canva=
s
>     is somewhat equivalent to dragging a "PenalizeFlowFile" processor ont=
o
>     the canvas (meaning I suggest the template is named PenalizeFlowFile)=
.
>
>     Regards,
>     Matt
>
>     On Wed, Aug 31, 2016 at 10:46 PM, Adam Taft <adam@adamtaft.com> wrote=
:
>     > In the wild west of HTTP response codes, a 500 Server Error could m=
ean
>     > practically anything.  In my experience, you can't infer any semant=
ic
>     > meaning for what a 500 status code could mean, unless you're very f=
amiliar
>     > with the server application.
>     >
>     > I'd even go so far as to suggest, if a modification is made to Post=
HTTP,
>     > that all non-200 response codes should be penalized.  The dataflow =
manager
>     > can always adjust the penalization timeout towards zero if a proces=
sing
>     > delay is not warranted.
>     >
>     > Unrelated, but this also reminds me, we really need a PenalizeFlowF=
ile
>     > processor, which would allow a dataflow manager to penalize a flowf=
ile
>     > anywhere that is deemed necessary, even if other processors haven't=
 done so
>     > (have routed to success).
>     >
>     >
>     > On Wed, Aug 31, 2016 at 1:54 PM, Andrew Grande <aperepel@gmail.com>=
 wrote:
>     >
>     >> Wasn't HTTP 400 Bad Request meant for that? 500 only means the ser=
ver
>     >> failed, not necessarily due to user input.
>     >>
>     >> Andrew
>     >>
>     >> On Wed, Aug 31, 2016, 10:16 AM Mark Payne <markap14@hotmail.com> w=
rote:
>     >>
>     >> > Hey Chris,
>     >> >
>     >> > I think it is reasonable to penalize when we receive a 500 respo=
nse. 500
>     >> > means Internal Server Error, and it is
>     >> > very reasonable to believe that the Internal Server Error occurr=
ed due to
>     >> > the specific input (i.e., that it may not
>     >> > always occur with different input). So penalizing the FlowFile s=
o that it
>     >> > can be retried after a little bit is reasonable
>     >> > IMO.
>     >> >
>     >> > When using the prioritizers, any FlowFile that is penalized will=
 not hold
>     >> > up other FlowFiles. They are always at the
>     >> > bottom of the queue until the penalization expires.
>     >> >
>     >> > Thanks
>     >> > -Mark
>     >> >
>     >> >
>     >> > > On Aug 31, 2016, at 10:06 AM, McDermott, Chris Kevin (MSDU -
>     >> > STaTS/StorefrontRemote) <chris.mcdermott@hpe.com> wrote:
>     >> > >
>     >> > > I wanted to ask if it would be at all sane to have the PostHTT=
P
>     >> > processor penalize a flowfile on 5xx response.  5xx indicates th=
at the
>     >> > request may be good but it cannot be handle by the server Curren=
tly it
>     >> > seems the processor routes files eliciting this response to the =
failure
>     >> > output but does not penalize them.  What do we think of adding s=
uch
>     >> > penalization?
>     >> > >
>     >> > > On a related note.  If a file penalized file is routed to a fu=
nnel that
>     >> > is connect to a processor via a connection with the OldestFlowFi=
leFirst
>     >> > prioritizer will the consumption of files from that connection b=
e blocked
>     >> > until penalization period is over?
>     >> > >
>     >> > > What I am trying to accomplish is this: I am using PostHTTP to=
 send
>     >> > files to web service that is throttling incoming data by returni=
ng a 500
>     >> > response.  When that happens I want to slow down files being to =
that that
>     >> > service.
>     >> > >
>     >> > > Thanks,
>     >> > >
>     >> > > Chris McDermott.
>     >> > >
>     >> > > Remote Business Analytics
>     >> > > STaTS/StoreFront Remote
>     >> > > HPE Storage
>     >> > > Hewlett Packard Enterprise
>     >> > > Mobile: +1 978-697-5315
>     >> > >
>     >> > >
>     >> >
>     >> >
>     >>
>
>