nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: NiFi 1.4 Clustering Error: Cannot Replicate Request GET /nifi-api/flow/current-user
Date Mon, 29 Jan 2018 17:28:38 GMT
Ryan,

I remember creating an issue for something that seems similar to what
you are running into:

https://issues.apache.org/jira/browse/NIFI-3642

Long story short, I believe you do need to specify a value for
nifi.web.http.host because that will be used to replicate requests
that come in to the REST API, so each node needs that value to be
something that is reachable by the other nodes.

-Bryan


On Mon, Jan 29, 2018 at 12:03 PM, Ryan H
<ryan.howell.development@gmail.com> wrote:
> Dev Team,
>
>
> I am running into an interesting issue while trying to cluster NiFi in a
> containerized environment (Docker containers running on DC/OS cluster) and I
> am somewhat stuck with what to do. I am starting with getting just 2 NiFi
> nodes with a single external zookeeper instance (just to get it working,
> will not use for production). Currently our DC/OS cluster does not support
> container-to-container communication (no overlay network support at the
> moment) so we are using VIP’s to expose required ports on the container(s)
> so traffic can be mapped to a well known address and correct container port
> even though the host/host port may change.
>
>
> Currently everything spins up and the UI can be accessed on whatever node is
> elected the Cluster Coordinator (in this case it is the Primary Node as
> well) (does show that there are 2 nodes in the cluster). However, any action
> taken on the canvas results in the following error shown in the UI:
>
>
> Node localhost:80 is unable to fulfill this request due to: Transaction
> c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress.
>
>
> When trying to access the UI of the other node, the canvas cannot be reached
> and the following error is shown on the error splash screen (increased read
> timeout to 30 secs, still the same):
>
>
> An unexpected error has occurred
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
>
>
> If configured to use the hostname of the container, then the error is:
>
> unknown host exception
>
>
> In the NiFi logs, the following errors are present (as well as some other
> warnings):
>
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to localhost:80 due to
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:641)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:852)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.lang.Thread.run(Thread.java:748)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> Caused by: java.net.SocketTimeoutException: Read timed out
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.net.SocketInputStream.socketRead0(Native Method)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
>
> 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut
> ... 12 common frames omitted
>
>
>
> 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4]
> o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message
> from <ip-host.aws.internal.address> due to java.io.EOFException
>
> 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut
> java.io.EOFException: null
>
> 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:112)
>
> 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142)
>
> 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
>
> 2018-01-29 15:19:42,828 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> 2018-01-29 15:19:42,828 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> 2018-01-29 15:19:42,828 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.lang.Thread.run(Thread.java:748)
>
> 2018-01-29 15:19:45,505 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 bytes)
> from localhost:80 in 2 millis
>
> 2018-01-29 15:19:46,118 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 bytes)
> from localhost:80 in 2 millis
>
> 2018-01-29 15:19:46,119 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1]
> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29
> 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29
> 15:19:46,119; send took 4 millis
>
> 2018-01-29 15:19:47,599 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in
> 7490 nanos
>
> 2018-01-29 15:19:50,515 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 bytes)
> from localhost:80 in 4 millis
>
> 2018-01-29 15:19:51,122 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 bytes)
> from localhost:80 in 2 millis
>
> 2018-01-29 15:19:51,123 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1]
> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29
> 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29
> 15:19:51,123; send took 4 millis
>
> 2018-01-29 15:19:52,601 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in
> 7776 nanos
>
> 2018-01-29 15:19:52,828 INFO [NiFi logging handler] org.apache.nifi.StdOut
> 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9]
> o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message
> from <ip-host.aws.internal.address> due to java.io.EOFException
>
> 2018-01-29 15:19:52,829 INFO [NiFi logging handler] org.apache.nifi.StdOut
> java.io.EOFException: null
>
>
>
> Here are relevant configs in nifi.properties file:
>
> # web properties #
>
> nifi.web.war.directory=./lib
>
> # for web host, I have tried: localhost, container hostname, left blank (to
> listen on all interfaces), the VIP configured for external access all of
> which have failed (read timeout or unknown host exception)
>
> nifi.web.http.host=
>
> nifi.web.http.port=80
>
> nifi.web.http.network.interface.default=
>
> nifi.web.https.host=
>
> nifi.web.https.port=
>
> nifi.web.https.network.interface.default=
>
> nifi.web.jetty.working.directory=./work/jetty
>
> nifi.web.jetty.threads=200
>
>
> # cluster node properties (only configure for cluster nodes) #
>
> nifi.cluster.is.node=true
>
> nifi.cluster.node.address=nifi-node1.my.vip.for.dcos
>
> nifi.cluster.node.protocol.port=9001
>
> nifi.cluster.node.protocol.threads=10
>
> nifi.cluster.node.protocol.max.threads=50
>
> nifi.cluster.node.event.history.size=25
>
> nifi.cluster.node.connection.timeout=30 sec
>
> nifi.cluster.node.read.timeout=30 sec
>
> nifi.cluster.node.max.concurrent.requests=100
>
> nifi.cluster.firewall.file=
>
> nifi.cluster.flow.election.max.wait.time=1 mins
>
> nifi.cluster.flow.election.max.candidates=
>
>
> # zookeeper properties, used for cluster management #
>
> nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181
>
> nifi.zookeeper.connect.timeout=3 secs
>
> nifi.zookeeper.session.timeout=3 secs
>
> nifi.zookeeper.root.node=/some-unique-id
>
>
>
> My suspicion why this is happening is due to the “nifi.web.http.host=”
> configuration does not have a discoverable address and I am not sure what I
> can configure this to, to make this work. I have configured a VIP for port
> 80 so that it can be externalized (and load balanced) and be reached outside
> of the DCOS cluster via web browser (which works for accessing the UI).  It
> seems that the UI replication relies on the web address of the other nodes
> (I was thinking that this was all done via the nifi.cluster.node.address and
> nifi.cluster.node.protocol.port properties, but I may have been wrong. What
> else is the value of “nifi.web.http.host=” used for?
>
>
> Any ideas or any clarity that can be shed on this would be greatly
> appreciated!
>
>
>
> Thanks in Advance,
>
>
> Ryan H.
>
>

Mime
View raw message