geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Barrett <jbarr...@pivotal.io>
Subject Re: Odg: Odg: Restart gateway-receiver
Date Tue, 26 Nov 2019 16:25:25 GMT
It’s is likely that your firewall is closing idle connections. This is very common for a
firewall to drop the state information for a connection that hasn’t seen any traffic in
some set period of time. It’s a way for firewalls to reclaim resources for connections that
were likely closed on one side or the other without any FIN sequence.

If you don’t have any control over your firewall then we should explore a keep alive method.

-Jake


> On Nov 26, 2019, at 2:06 AM, Mario Kevo <mario.kevo@est.tech> wrote:
> 
> Thanks a lot @Barry Oglesby<mailto:boglesby@pivotal.io>!
> 
> It seems that this closing inactive connection is done by Kubernetes as we run Geode
on it.
> 
> BR,
> Mario
> 
> ________________________________
> Šalje: Barry Oglesby <boglesby@pivotal.io>
> Poslano: 22. studenog 2019. 22:35
> Prima: Mario Kevo <mario.kevo@est.tech>
> Kopija: dev@geode.apache.org <dev@geode.apache.org>
> Predmet: Re: Odg: Restart gateway-receiver
> 
>> If we don't send any event, the connection will be closed after some time as connection
is inactive.
> 
> Are you seeing this behavior? I don't think this is true by default.
> 
> AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory:
> 
> pf.setReadTimeout(this.socketReadTimeout);
> pf.setIdleTimeout(connectionIdleTimeOut);
> 
> By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is -1 (disabled).
> 
> Each Event Processor thread will have its own connection to the remote site:
> 
> Event Processor for GatewaySender_ny.1: GatewaySenderEventRemoteDispatcher.initializeConnection
connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@306907760
> Event Processor for GatewaySender_ny.2: GatewaySenderEventRemoteDispatcher.initializeConnection
connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@608855137
> Event Processor for GatewaySender_ny.0: GatewaySenderEventRemoteDispatcher.initializeConnection
connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@950613560
> Event Processor for GatewaySender_ny.4: GatewaySenderEventRemoteDispatcher.initializeConnection
connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@1005378489
> Event Processor for GatewaySender_ny.3: GatewaySenderEventRemoteDispatcher.initializeConnection
connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@629246640
> 
> There will be one Event Processor thread for each dispatcher thread (5 by default).
> 
> There aren't any good public ways to monitor the connections other than JMX.
> 
> One way to monitor these connections is with ConnectionStats on the sender side.
> 
> You can do that with vsd: ClientStats -> GatewaySenderStats -> connections
> 
> You can also do it with code like:
> 
> private int getConnectionCount(String gatewaySenderId) {
>  AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId);
>  int totalConnections = 0;
>  if (sender != null) {
>    for (ConnectionStats connectionStats : sender.getProxy().getEndpointManager().getAllStats().values())
{
>      totalConnections += connectionStats.getConnections();
>    }
>    System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + totalConnections);
>  }
>  return totalConnections;
> }
> 
> You can also dump whether the dispatcher is connected like:
> 
> private void dumpConnected(String gatewaySenderId) {
>  AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId);
>  if (sender.isParallel()) {
>    ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = (ConcurrentParallelGatewaySenderEventProcessor)
sender.getEventProcessor();
>    for (ParallelGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors())
{
>      System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote());
>    }
>  } else {
>    ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = (ConcurrentSerialGatewaySenderEventProcessor)
sender.getEventProcessor();
>    List<SerialGatewaySenderEventProcessor> processors = concurrentProcessor.getProcessors();
>    for (SerialGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors())
{
>      System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote());
>    }
>  }
> }
> 
> The isConnectedToRemote method does:
> 
> return connection != null && !connection.isDestroyed();
> 
> Thanks,
> Barry Oglesby
> 
> 
> 
> On Thu, Nov 21, 2019 at 11:15 PM Mario Kevo <mario.kevo@est.tech> wrote:
> Hi,
> 
> @Barry Oglesby<mailto:boglesby@pivotal.io>, thanks for the clarification.
> 
> If we don't send any event, the connection will be closed after some time as connection
is inactive.
> Is it possible to somehow monitor from the app if the replication is established to get
information if there is a some problem with network or it is just closed due to inactivity?
> Can we monitor the replication on some other way than looking "isConnected" state on
JMX?
> 
> BR,
> Mario
> ________________________________
> Šalje: Barry Oglesby <boglesby@pivotal.io<mailto:boglesby@pivotal.io>>
> Poslano: 14. studenog 2019. 18:29
> Prima: dev@geode.apache.org<mailto:dev@geode.apache.org> <dev@geode.apache.org<mailto:dev@geode.apache.org>>
> Predmet: Re: Restart gateway-receiver
> 
> Mario,
> 
> Thats the current behavior. When the sender is initially started, it
> attempts to connect to the receiver. If it does not connect, it won't retry
> until there is a batch of events to send. Look for callers of
> GatewaySenderEventRemoteDispatcher.initializeConnection to see the
> behavior. It could be changed to have a task to retry lost connections, but
> generally there are events in the queue, so the connection is
> re-established pretty quickly by the event processor thread.
> 
> Thanks,
> Barry Oglesby
> 
> 
> 
>> On Wed, Nov 13, 2019 at 4:55 AM Mario Kevo <mario.kevo@est.tech> wrote:
>> 
>> Hi geode dev,
>> 
>> After creating gateways senders and receivers between two geode clusters
>> replications is established. After restart gateway receiver, sender will
>> not connect to it until we send some entry from sender to receiver.
>> Is this a normal behavior or a bug?
>> Should geode have some mechanism for checking if connection is established
>> no matter if entry is sent or not?
>> 
>> BR,
>> Mario
>> 

Mime
View raw message