nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Percivall <joeperciv...@gmail.com>
Subject Re: InvokeHTTP and SocketTimeoutException
Date Fri, 27 Jan 2017 15:12:09 GMT
Hello Giovanni,

You are definitely right that the docs currently do not match what actually
happens in some cases. I've created a ticket to address it here:

https://issues.apache.org/jira/browse/NIFI-3408

Joe

On Thu, Jan 26, 2017 at 11:23 AM, Giovanni Lanzani <
giovannilanzani@godatadriven.com> wrote:

> Hi Joe,
>
>
>
> Also SocketTimeoutException (connection made, but no response sent within
> the timeout) no attributes are added. What I’m confused about is that in
> the Failure case docs, it was specified that information over the failure
> would be added.
>
>
>
> As SocketTimeoutException and ConnectException indicate two different
> failure mode, it would be helpful (at least for me, but let me know
> otherwise) if NiFi would write in some attribute the (Java) exception that
> took place.
>
>
>
> Downstream, using RouteOnAttribute or a conditional UpdateAttribute,
> different actions can be taken (if the server was down you might want to
> add a 5’ penalty, if it was busy a 30’ penalty, for example).
>
>
>
> Cheers,
>
>
>
> Giovanni
>
>
>
> *From:* Joe Percivall [mailto:jpercivall@apache.org]
> *Sent:* Thursday, January 26, 2017 3:48 PM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: InvokeHTTP and SocketTimeoutException
>
>
>
> Hello Giovanni,
>
>
>
> The InvokeHTTP processor only adds attributes in the event that the
> request actually gets a response. This is primarily due to the fact that
> the attributes that are written (status codes, status message, etc.) are
> only available if we get back a response. The exception you're hitting is
> due to failing to connect to the server at all (and thus no status codes or
> messages).
>
>
>
> Is there a specific attribute value that you would like to be added in the
> event of a hard failure like this?
>
>
>
> Cheers,
>
> Joe
>
>
>
> On Thu, Jan 26, 2017 at 3:23 AM, Giovanni Lanzani <
> giovannilanzani@godatadriven.com> wrote:
>
> Hi Andy,
>
>
>
> Thanks for chiming in. I’ve gathered some info down here:
>
>
>
> The LogAttribute before InvokeHTTP:
>
>
>
> 2017-01-26 09:15:10,373 INFO [Timer-Driven Process Thread-4]
> o.a.n.processors.standard.LogAttribute LogAttribute[id=d9d45caf-0159-
> 1000-d4a6-5127a
>
> fdeaf64] logging for flow file StandardFlowFileRecord[uuid=dffe5b
> 52-b5f1-4597-8540-dc0baaff29ad,claim=StandardContentClaim
> [resourceClaim=Standar
>
> dResourceClaim[id=1485197201540-1, container=default, section=1],
> offset=37389, length=10],offset=0,name=258752766597833,size=10]
>
> --------------------------------------------------
>
> Standard FlowFile Attributes
>
> Key: 'entryDate'
>
>         Value: 'Thu Jan 26 09:14:57 CET 2017'
>
> Key: 'lineageStartDate'
>
>         Value: 'Thu Jan 26 09:14:57 CET 2017'
>
> Key: 'fileSize'
>
>         Value: '10'
>
> FlowFile Attribute Map Content
>
> Key: 'filename'
>
>         Value: '258752766597833'
>
> Key: 'path'
>
>         Value: './'
>
> Key: 'uuid'
>
>         Value: 'dffe5b52-b5f1-4597-8540-dc0baaff29ad'
>
> --------------------------------------------------
>
>
>
> After:
>
>
>
> 2017-01-26 09:15:48,572 INFO [Timer-Driven Process Thread-6]
> o.a.n.processors.standard.LogAttribute LogAttribute[id=d9d49cf0-0159-1000-c5cd-c0810449e070]
> logging for flow file StandardFlowFileRecord[uuid=dffe5b52-b5f1-4597-8540-
> dc0baaff29ad,claim=StandardContentClaim [resourceClaim=
> StandardResourceClaim[id=1485197201540-1, container=default, section=1],
> offset=37389, length=10],offset=0,name=258752766597833,size=10]
>
> --------------------------------------------------
>
> Standard FlowFile Attributes
>
> Key: 'entryDate'
>
>         Value: 'Thu Jan 26 09:14:57 CET 2017'
>
> Key: 'lineageStartDate'
>
>         Value: 'Thu Jan 26 09:14:57 CET 2017'
>
> Key: 'fileSize'
>
>         Value: '10'
>
> FlowFile Attribute Map Content
>
> Key: 'filename'
>
>         Value: '258752766597833'
>
> Key: 'path'
>
>         Value: './'
>
> Key: 'uuid'
>
>         Value: 'dffe5b52-b5f1-4597-8540-dc0baaff29ad'
>
> --------------------------------------------------
>
>
>
> The exception InvokeHTTP is raising (in this case a ConnectException, I
> can make it a SocketTimeout with some extra work but I think new attributes
> should be present in this case as well)
>
>
>
> 2017-01-26 09:15:10,975 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent
> Stopped scheduling LogAttribute[id=d9d45caf
>
> -0159-1000-d4a6-5127afdeaf64] to run
>
> 2017-01-26 09:15:11,100 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controll
>
> er.FlowController@68275864 // Another save pending = true
>
> 2017-01-26 09:15:11,955 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controll
>
> er.FlowController@68275864 // Another save pending = false
>
> 2017-01-26 09:15:16,228 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent
> Scheduled InvokeHTTP[id=d9d48225-0159-1000-
>
> 9958-4bed80de9fa6] to run with 1 threads
>
> 2017-01-26 09:15:16,842 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controll
>
> er.FlowController@68275864 // Another save pending = false
>
> 2017-01-26 09:15:17,551 ERROR [Timer-Driven Process Thread-8]
> o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=d9d48225-0159-
> 1000-9958-4bed8
>
> 0de9fa6] Routing to Failure due to exception: java.net.ConnectException:
> Failed to connect to /127.0.0.1:8000: java.net.ConnectException: Failed
>
> to connect to /127.0.0.1:8000
>
> 2017-01-26 09:15:17,553 ERROR [Timer-Driven Process Thread-8]
> o.a.nifi.processors.standard.InvokeHTTP
>
> java.net.ConnectException: Failed to connect to /127.0.0.1:8000
>
>         at com.squareup.okhttp.internal.io.RealConnection.
> connectSocket(RealConnection.java:139) ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:108)
> ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.internal.http.StreamAllocation.
> findConnection(StreamAllocation.java:184) ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.internal.http.StreamAllocation.
> findHealthyConnection(StreamAllocation.java:126) ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.internal.http.StreamAllocation.
> newStream(StreamAllocation.java:95) ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:283)
> ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
> ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.Call.getResponse(Call.java:286)
> ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
> ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
> ~[okhttp-2.7.1.jar:na]
>
>         at com.squareup.okhttp.Call.execute(Call.java:80)
> ~[okhttp-2.7.1.jar:na]
>
>         at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:624)
> ~[nifi-standard-processors-1.0.0.jar:1.0.0]
>
>         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.0.0.jar:1.0.0]
>
>         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0]
>
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.
> jar:1.0.0]
>
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.
> jar:1.0.0]
>
>         at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_77]
>
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_77]
>
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_77]
>
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_77]
>
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_77]
>
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_77]
>
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
>
>
>
>
>
> The InvokeHTTP config is standard except:
>
>
>
> HTTP Method: POST
>
> URL 127.0.0.1:8000 (nothing is running there)
>
> Read timeout: 5 secs
>
> Content-type: text/plain
>
>
>
> The test flow is very simple, the real one is more complicated.
>
>
>
>
>
>
>
>
>
> *From:* Andy LoPresto [mailto:alopresto@apache.org]
> *Sent:* Wednesday, January 25, 2017 10:47 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: InvokeHTTP and SocketTimeoutException
>
>
>
> Giovanni,
>
>
>
> You are correct; the flowfile should have new attributes
> (invokehttp.status.code, etc.) on failure. Can you put a LogAttribute
> processor before and after the InvokeHTTP processor and print all of the
> attributes of the flowfile, then copy them here? If you can also share the
> configuration of your InvokeHTTP processor, that will help us diagnose the
> issue. Thanks.
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Jan 25, 2017, at 7:44 AM, Giovanni Lanzani <
> giovannilanzani@godatadriven.com> wrote:
>
>
>
> Hi all,
>
> I'm using InvokeHTTP to do a POST request to a server. I'm trying to
> simulate the server accepting connection but NOT giving response within the
> limit I set in NiFi.
>
> The processor fails with SocketTimeoutException and the FlowFile is
> rerouted to the Failure relationship (up until this: all is good!).
>
> The issue is that the exception was not written in the attributes of the
> FlowFile. No attributes are modified when the failure is occurring. By
> reading the docs for the failure relationship, I was under the impression
> that:
>
> "The original FlowFile [...] will have new attributes detailing the
> request".
>
> But I think that maybe that is not correct?! (I'm using NiFi 1.0.0)
>
> Thanks in advance for everybody that might chime in! Sorry in advance if
> this was a super dump question like my last one.
>
> Cheers,
>
> Giovanni
>
>
>
>
>
>
>
> --
>
> *Joe Percivall*
>
> linkedin.com/in/Percivall
>
> e: jpercivall@apache.com
>



-- 
*Joe Percivall*
e: joepercivall@gmail.com

Mime
View raw message