nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan H <ryan.howell.developm...@gmail.com>
Subject Re: NiFi 1.4 Clustering Error: Cannot Replicate Request GET /nifi-api/flow/current-user
Date Mon, 29 Jan 2018 20:03:52 GMT
Good deal, thanks for getting back to me about it. I think that I may make
some changes and potentially open a PR depending on what comes of it (I'll
have to take a look at the workflow and all of that good stuff first).


Cheers,

Ryan H

On Mon, Jan 29, 2018 at 12:55 PM, Bryan Bende <bbende@gmail.com> wrote:

> Ryan,
>
> I'm not that familiar with Docker and DCOS, but I think what you said
> is correct...
>
> The issue is that you currently can't leave nifi.web.http.host blank
> because that will cause the "node API address" of each node to be
> calculated as 'localhost', which then means replication of requests
> fails.
>
> So you have to set nifi.web.http.host to something that each node can
> reach. I'm not familiar enough with your setup to know if there is a
> way to do that.
>
> -Bryan
>
>
> On Mon, Jan 29, 2018 at 12:39 PM, Ryan H
> <ryan.howell.development@gmail.com> wrote:
> > Hi Bryan,
> >
> > Yes that makes total sense, and it is what I figured was happening. So
> > whatever is configured for nifi.web.http.host is where api calls will go
> to,
> > but this is also what jetty will bind to, correct? So in my case, I would
> > have to have the additional property mentioned in
> > https://issues.apache.org/jira/browse/NIFI-3642 since jetty can't bind
> to a
> > VIP.
> >
> > -Ryan H
> >
> > On Mon, Jan 29, 2018 at 12:28 PM, Bryan Bende <bbende@gmail.com> wrote:
> >>
> >> Ryan,
> >>
> >> I remember creating an issue for something that seems similar to what
> >> you are running into:
> >>
> >> https://issues.apache.org/jira/browse/NIFI-3642
> >>
> >> Long story short, I believe you do need to specify a value for
> >> nifi.web.http.host because that will be used to replicate requests
> >> that come in to the REST API, so each node needs that value to be
> >> something that is reachable by the other nodes.
> >>
> >> -Bryan
> >>
> >>
> >> On Mon, Jan 29, 2018 at 12:03 PM, Ryan H
> >> <ryan.howell.development@gmail.com> wrote:
> >> > Dev Team,
> >> >
> >> >
> >> > I am running into an interesting issue while trying to cluster NiFi
> in a
> >> > containerized environment (Docker containers running on DC/OS cluster)
> >> > and I
> >> > am somewhat stuck with what to do. I am starting with getting just 2
> >> > NiFi
> >> > nodes with a single external zookeeper instance (just to get it
> working,
> >> > will not use for production). Currently our DC/OS cluster does not
> >> > support
> >> > container-to-container communication (no overlay network support at
> the
> >> > moment) so we are using VIP’s to expose required ports on the
> >> > container(s)
> >> > so traffic can be mapped to a well known address and correct container
> >> > port
> >> > even though the host/host port may change.
> >> >
> >> >
> >> > Currently everything spins up and the UI can be accessed on whatever
> >> > node is
> >> > elected the Cluster Coordinator (in this case it is the Primary Node
> as
> >> > well) (does show that there are 2 nodes in the cluster). However, any
> >> > action
> >> > taken on the canvas results in the following error shown in the UI:
> >> >
> >> >
> >> > Node localhost:80 is unable to fulfill this request due to:
> Transaction
> >> > c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress.
> >> >
> >> >
> >> > When trying to access the UI of the other node, the canvas cannot be
> >> > reached
> >> > and the following error is shown on the error splash screen (increased
> >> > read
> >> > timeout to 30 secs, still the same):
> >> >
> >> >
> >> > An unexpected error has occurred
> >> > com.sun.jersey.api.client.ClientHandlerException:
> >> > java.net.SocketTimeoutException: Read timed out
> >> >
> >> >
> >> > If configured to use the hostname of the container, then the error is:
> >> >
> >> > unknown host exception
> >> >
> >> >
> >> > In the NiFi logs, the following errors are present (as well as some
> >> > other
> >> > warnings):
> >> >
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3]
> >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> >> > GET
> >> > /nifi-api/flow/current-user to localhost:80 due to
> >> > com.sun.jersey.api.client.ClientHandlerException:
> >> > java.net.SocketTimeoutException: Read timed out
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3]
> >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > com.sun.jersey.api.client.ClientHandlerException:
> >> > java.net.SocketTimeoutException: Read timed out
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.
> handle(URLConnectionClientHandler.java:155)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at com.sun.jersey.api.client.Client.handle(Client.java:652)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(
> GZIPContentEncodingFilter.java:123)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at com.sun.jersey.api.client.WebResource.access$200(
> WebResource.java:74)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> > com.sun.jersey.api.client.WebResource$Builder.get(
> WebResource.java:509)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > org.apache.nifi.cluster.coordination.http.replication.
> ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.
> java:641)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > org.apache.nifi.cluster.coordination.http.replication.
> ThreadPoolRequestReplicator$NodeHttpRequest.run(
> ThreadPoolRequestReplicator.java:852)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> > java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.lang.Thread.run(Thread.java:748)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > Caused by: java.net.SocketTimeoutException: Read timed out
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.net.SocketInputStream.socketRead0(Native Method)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.net.SocketInputStream.read(SocketInputStream.java:171)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> HttpURLConnection.java:1587)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> HttpURLConnection.java:1492)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> > java.net.HttpURLConnection.getResponseCode(
> HttpURLConnection.java:480)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._
> invoke(URLConnectionClientHandler.java:253)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.
> handle(URLConnectionClientHandler.java:153)
> >> >
> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > ... 12 common frames omitted
> >> >
> >> >
> >> >
> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4]
> >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol
> message
> >> > from <ip-host.aws.internal.address> due to java.io.EOFException
> >> >
> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > java.io.EOFException: null
> >> >
> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.
> unmarshal(JaxbProtocolContext.java:112)
> >> >
> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.
> dispatchRequest(SocketProtocolListener.java:142)
> >> >
> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> > org.apache.nifi.io.socket.SocketListener$2$1.run(
> SocketListener.java:136)
> >> >
> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> >> >
> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at
> >> >
> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> >> >
> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > at java.lang.Thread.run(Thread.java:748)
> >> >
> >> > 2018-01-29 15:19:45,505 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5]
> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> >> > 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555
> bytes)
> >> > from localhost:80 in 2 millis
> >> >
> >> > 2018-01-29 15:19:46,118 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6]
> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> >> > f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555
> bytes)
> >> > from localhost:80 in 2 millis
> >> >
> >> > 2018-01-29 15:19:46,119 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1]
> >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29
> >> > 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at
> 2018-01-29
> >> > 15:19:46,119; send took 4 millis
> >> >
> >> > 2018-01-29 15:19:47,599 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1]
> >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2
> heartbeats in
> >> > 7490 nanos
> >> >
> >> > 2018-01-29 15:19:50,515 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7]
> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> >> > d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555
> bytes)
> >> > from localhost:80 in 4 millis
> >> >
> >> > 2018-01-29 15:19:51,122 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8]
> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> >> > 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555
> bytes)
> >> > from localhost:80 in 2 millis
> >> >
> >> > 2018-01-29 15:19:51,123 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1]
> >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29
> >> > 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at
> 2018-01-29
> >> > 15:19:51,123; send took 4 millis
> >> >
> >> > 2018-01-29 15:19:52,601 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1]
> >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2
> heartbeats in
> >> > 7776 nanos
> >> >
> >> > 2018-01-29 15:19:52,828 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9]
> >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol
> message
> >> > from <ip-host.aws.internal.address> due to java.io.EOFException
> >> >
> >> > 2018-01-29 15:19:52,829 INFO [NiFi logging handler]
> >> > org.apache.nifi.StdOut
> >> > java.io.EOFException: null
> >> >
> >> >
> >> >
> >> > Here are relevant configs in nifi.properties file:
> >> >
> >> > # web properties #
> >> >
> >> > nifi.web.war.directory=./lib
> >> >
> >> > # for web host, I have tried: localhost, container hostname, left
> blank
> >> > (to
> >> > listen on all interfaces), the VIP configured for external access all
> of
> >> > which have failed (read timeout or unknown host exception)
> >> >
> >> > nifi.web.http.host=
> >> >
> >> > nifi.web.http.port=80
> >> >
> >> > nifi.web.http.network.interface.default=
> >> >
> >> > nifi.web.https.host=
> >> >
> >> > nifi.web.https.port=
> >> >
> >> > nifi.web.https.network.interface.default=
> >> >
> >> > nifi.web.jetty.working.directory=./work/jetty
> >> >
> >> > nifi.web.jetty.threads=200
> >> >
> >> >
> >> > # cluster node properties (only configure for cluster nodes) #
> >> >
> >> > nifi.cluster.is.node=true
> >> >
> >> > nifi.cluster.node.address=nifi-node1.my.vip.for.dcos
> >> >
> >> > nifi.cluster.node.protocol.port=9001
> >> >
> >> > nifi.cluster.node.protocol.threads=10
> >> >
> >> > nifi.cluster.node.protocol.max.threads=50
> >> >
> >> > nifi.cluster.node.event.history.size=25
> >> >
> >> > nifi.cluster.node.connection.timeout=30 sec
> >> >
> >> > nifi.cluster.node.read.timeout=30 sec
> >> >
> >> > nifi.cluster.node.max.concurrent.requests=100
> >> >
> >> > nifi.cluster.firewall.file=
> >> >
> >> > nifi.cluster.flow.election.max.wait.time=1 mins
> >> >
> >> > nifi.cluster.flow.election.max.candidates=
> >> >
> >> >
> >> > # zookeeper properties, used for cluster management #
> >> >
> >> > nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181
> >> >
> >> > nifi.zookeeper.connect.timeout=3 secs
> >> >
> >> > nifi.zookeeper.session.timeout=3 secs
> >> >
> >> > nifi.zookeeper.root.node=/some-unique-id
> >> >
> >> >
> >> >
> >> > My suspicion why this is happening is due to the “nifi.web.http.host=”
> >> > configuration does not have a discoverable address and I am not sure
> >> > what I
> >> > can configure this to, to make this work. I have configured a VIP for
> >> > port
> >> > 80 so that it can be externalized (and load balanced) and be reached
> >> > outside
> >> > of the DCOS cluster via web browser (which works for accessing the
> UI).
> >> > It
> >> > seems that the UI replication relies on the web address of the other
> >> > nodes
> >> > (I was thinking that this was all done via the
> nifi.cluster.node.address
> >> > and
> >> > nifi.cluster.node.protocol.port properties, but I may have been
> wrong.
> >> > What
> >> > else is the value of “nifi.web.http.host=” used for?
> >> >
> >> >
> >> > Any ideas or any clarity that can be shed on this would be greatly
> >> > appreciated!
> >> >
> >> >
> >> >
> >> > Thanks in Advance,
> >> >
> >> >
> >> > Ryan H.
> >> >
> >> >
> >
> >
>

Mime
View raw message