tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raúl García <rgar...@emovilia.com>
Subject RE: Tomcat 6 - Cluster error.
Date Fri, 21 Dec 2007 10:19:49 GMT
Hi,
We continue having problems.
Same situation as yesterday, timeouts and Faulty members, node 1 reports
that node 2 is not alive (But it is!!!)
The error at node1:

CATALINA.OUT (Node_1)
=====================
Dec 21, 2007 8:27:33 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
        at [...]
Dec 21, 2007 8:27:33 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNING: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:40
02,localhost,4002, alive=65950380,id={-4 38 19 -4 -3 -109 79 -79 -72 -13 99
-57 37 -60 19 -63 }, payload={}, command={}, domain={}, ]
Dec 21, 2007 8:27:33 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
localhost,4002, alive=65950380,id={-4 38 19 -4 -3 -109 79 -79 -72 -13 99 -57
37 -60 19 -63 }, payload={}, command={}, domain={}, ]
Dec 21, 2007 8:27:47 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
alhost:4002,localhost,4002, alive=65963405,id={-4 38 19 -4 -3 -109 79 -79
-72 -13 99 -57 37 -60 19 -63 }, payload={}, command={}, domain={}, ]]
message. Will verify.
Dec 21, 2007 8:27:47 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
localhost,4002, alive=65963405,id={-4 38 19 -4 -3 -109 79 -79 -72 -13 99 -57
37 -60 19 -63 }, payload={}, command={}, domain={}, ]]
Dec 21, 2007 8:27:47 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
ms.).; Faulty members:tcp://localhost:4002;
        at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
lelNioSender.java:97)
        at [...]
Dec 21, 2007 8:27:49 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
:4002,localhost,4002, alive=65966764,id={-4 38 19 -4 -3 -109 79 -79 -72 -13
99 -57 37 -60 19 -63 }, payload={}, command={66 65 66 89 45 65 76 69 88
...(9)}, domain={}, ]]
Dec 21, 2007 8:27:49 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost
:4002,localhost,4002, alive=65966764,id={-4 38 19 -4 -3 -109 79 -79 -72 -13
99 -57 37 -60 19 -63 }, payload={}, command={66 65 66 89 45 65 76 69 88
...(9)}, domain={}, ]
Dec 21, 2007 8:27:53 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
        at [...]
Dec 21, 2007 8:27:53 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
        at [...]
Dec 21, 2007 8:27:54 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
        at [...]
Dec 21, 2007 8:27:54 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
alhost:4002,localhost,4002, alive=65966411,id={-4 38 19 -4 -3 -109 79 -79
-72 -13 99 -57 37 -60 19 -63 }, payload={}, command={}, domain={}, ]]
message. Will verify.
Dec 21, 2007 8:27:54 AM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost:4002,
localhost,4002, alive=65966411,id={-4 38 19 -4 -3 -109 79 -79 -72 -13 99 -57
37 -60 19 -63 }, payload={}, command={}, domain={}, ]]
Dec 21, 2007 8:27:54 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(60000
ms.).; Faulty members:tcp://localhost:4002;
        at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
lelNioSender.java:97)
        at [...]

Now I paste my full server.xml configuration here, I don’t really know why
this is happening

NODE1:
======
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8006" shutdown="SHUTDOWN" debug="0">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <!--<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />-->
  <!--Initialize Jasper prior to webapps are loaded. Documentation at
/docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
  <!-- JMX Support for the Tomcat server. Documentation at
/docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
  <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Environment name="InstanceName" type="java.lang.String" value="pro1"/>

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more
named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    -->


    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking &
non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <!-- <Connector port="8080" protocol="HTTP/1.1" -->
    <!-- <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />-->

    <Connector port="8081" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
emptySessionPath="true"
               maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
               enableLookups="false" redirectPort="81443" acceptCount="1000"
               debug="0" connectionTimeout="20000"
disableUploadTimeout="true"
               compression="on"
                           compressionMinSize="2048"
                           noCompressionUserAgents="gozilla, traviata"
                           compressableMimeType="text/html,text/xml" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
-->


    <!-- An Engine represents the entry point (within Catalina) that
processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes
them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost" debug="0"
jvmRoute="PR1">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

                        <Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                 channelSendOptions="6">

<!--
                <Manager
className="org.apache.catalina.ha.session.BackupManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"
                   mapSendOptions="6"/>
-->

          <Manager className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>

          <Channel
className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership
className="org.apache.catalina.tribes.membership.McastService"
                        address="228.0.0.4"
                        port="45564"
                        frequency="1000"
                        dropTime="30000"/>
            <Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="127.0.0.1"
                      port="4001"
                      autoBind="100"
                      selectorTimeout="100"
                      maxThreads="12"/>

            <Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"/>
            </Sender>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
terceptor"/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
or"/>
          </Channel>

          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
 
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
          <!-- <Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->

          <Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/>

          <!-- <ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
-->
          <ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             debug="0" resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost" debug="0" appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web
applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="localhost_access_log." suffix=".txt" pattern="common"
resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

NODE2:
======
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8007" shutdown="SHUTDOWN" debug="0">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <!--<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />-->
  <!--Initialize Jasper prior to webapps are loaded. Documentation at
/docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" debug="0"/>
  <!-- JMX Support for the Tomcat server. Documentation at
/docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
  <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Environment name="InstanceName" type="java.lang.String" value="pro2"/>

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml"/>
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more
named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    -->


    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking &
non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <!-- <Connector port="8080" protocol="HTTP/1.1" -->
    <!-- <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />-->

    <Connector port="8082" protocol="HTTP/1.1" maxHttpHeaderSize="8192"
emptySessionPath="true"
               maxThreads="150" minSpareThreads="100" maxSpareThreads="300"
               enableLookups="false" redirectPort="82443" acceptCount="1000"
               debug="0" connectionTimeout="20000"
disableUploadTimeout="true"
               compression="on"
                           compressionMinSize="2048"
                           noCompressionUserAgents="gozilla, traviata"
                           compressableMimeType="text/html,text/xml" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
-->


    <!-- An Engine represents the entry point (within Catalina) that
processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes
them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost" debug="0"
jvmRoute="PR2">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

                        <Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                 channelSendOptions="6">

<!--
                <Manager
className="org.apache.catalina.ha.session.BackupManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"
                   mapSendOptions="6"/>
-->

          <Manager className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>

          <Channel
className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership
className="org.apache.catalina.tribes.membership.McastService"
                        address="228.0.0.4"
                        port="45564"
                        frequency="1000"
                        dropTime="30000"/>
            <Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="127.0.0.1"
                      port="4002"
                      autoBind="100"
                      selectorTimeout="100"
                      maxThreads="12"/>

            <Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"/>
            </Sender>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15In
terceptor"/>
            <Interceptor
className="org.apache.catalina.tribes.group.interceptors.ThroughputIntercept
or"/>
          </Channel>

          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
 
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
          <!-- <Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> -->

          <Deployer
className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/>

          <!-- <ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
-->
          <ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>


      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase" debug="0"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost" debug="0" appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web
applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="localhost_access_log." suffix=".txt" pattern="common"
resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>
==================


We notice that the cluster listener JvmRouteSessionIDBinderListener was not
activated.
          <!-- <ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
-->
So now we start both nodes uncommenting this line.

And we got this new error:

21-dic-2007 9:43:09 org.apache.catalina.core.ContainerBase backgroundProcess
WARNING: Exception processing manager
org.apache.catalina.ha.session.DeltaManager@1d9600d background process
java.lang.NullPointerException
        at
org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:682)
        at
org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:6
67)
        at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:
1316)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processC
hildren(ContainerBase.java:1601)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processC
hildren(ContainerBase.java:1610)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processC
hildren(ContainerBase.java:1610)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
ainerBase.java:1590)
        at java.lang.Thread.run(Thread.java:619)



I'm sorry for the big post, we are trying to configure the server since 15
days ago without luck. We are really desperate...

Any Idea o suggestion will be appreciated.

Best Regards, and thanks for your helpful time.
Raúl.




-----Mensaje original-----
De: Raúl García [mailto:rgarcia@emovilia.com] 
Enviado el: jueves, 20 de diciembre de 2007 15:45
Para: 'Tomcat Developers List'
Asunto: RE: Tomcat 6 - Cluster error.

Hi again,

Now we added this timeout config. attribute to the sender:

<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
 <Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="60000"/>
</Sender>

It works!, now the 2 nodes get nice replication, no more errors about that
Faulty members.

We use "pen" load balancer to redirect between the two nodes.
Now we got this error when we shutdown one of the nodes:

20-dic-2007 9:31:24 org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
        at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
        at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
        at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
er.java:814)
        at
org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:586)
        at
org.apache.catalina.ha.session.DeltaManager.sessionExpired(DeltaManager.java
:1211)
        at
org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:366)
        at
org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:352)
        at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:660)
        at
org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:
1111)
        at
org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSession
Facade.java:150)
        at pad.kernel.Resolver.service(Resolver.java:623)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
        at pad.kernel.EntryPointFilter.doFilter(EntryPointFilter.java:365)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:219)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:175)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
        at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.jav
a:269)
        at
org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
        at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347
)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:584)
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)

The error repeat and repeat forever until we re-startup the node. 
It's normal?


But once started up again, the replication is successful and the server
seems that still works. We will see the next hours.

Thank you very very much for your time.

Best Regards
Raúl.


-----Mensaje original-----
De: Filip Hanik - Dev Lists [mailto:devlists@hanik.com] 
Enviado el: miércoles, 19 de diciembre de 2007 19:03
Para: Tomcat Developers List
Asunto: Re: Tomcat 6 - Cluster error.

you have a 3sec timeout on your message send

SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
ms.).; Faulty members:tcp://localhost.localdomain:4002;

you can change this value in <Transport>
http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-sender.html

Filip

Raúl García wrote:
> Thanks Peter for the quick response,
>
> Now we are trying this config at both instances(nodes):
>
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>  <Membership
className="org.apache.catalina.tribes.membership.McastService"
>                          address="228.0.0.8"
>                          port="45578"
>                          frequency="1000"
>                          dropTime="30000"/>
>
>
> dropTime="30000", as you suggest.
>
> To reproduce the errors, now we are doing some stress massive hits at our
> local backup server (same config, same machine).
> Before a 120000 hits (java log-in simple service) during 1 hour (From
14:00
> to 15:00 aprox.), the instances log drop an error, but both instances seem
> to be alive.
>
> They returned the following (only once) error:
>
> ERRORS catalina.out:
> =================
>
> INSTANCE 1
> ================
> 19-dic-2007 14:37:45
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:29817 messages
>         Sent:17,71 MB (total)
>         Sent:17,71 MB (application)
>         Time:53,51 seconds
>         Tx Speed:0,33 MB/sec (total)
>         TxSpeed:0,33 MB/sec (application)
>         Error Msg:0
>         Rx Msg:30000 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:17,76 MB]
> 19-dic-2007 14:38:04
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:30000 messages
>         Sent:17,82 MB (total)
>         Sent:17,82 MB (application)
>         Time:53,73 seconds
>         Tx Speed:0,33 MB/sec (total)
>         TxSpeed:0,33 MB/sec (application)
>         Error Msg:0
>         Rx Msg:30204 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:17,88 MB]
> 19-dic-2007 14:52:03
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:39465 messages
>         Sent:23,42 MB (total)
>         Sent:23,42 MB (application)
>         Time:68,87 seconds
>         Tx Speed:0,34 MB/sec (total)
>         TxSpeed:0,34 MB/sec (application)
>         Error Msg:0
>         Rx Msg:40000 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:23,60 MB]
> 19-dic-2007 14:54:09
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:40000 messages
>         Sent:23,73 MB (total)
>         Sent:23,73 MB (application)
>         Time:69,61 seconds
>         Tx Speed:0,34 MB/sec (total)
>         TxSpeed:0,34 MB/sec (application)
>         Error Msg:0
>         Rx Msg:40491 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:23,87 MB]
> 19-dic-2007 15:15:18
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
>
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
> alhost.localdomain:4002,localhost.localdomain,4002, alive=5097079,id={73
-17
> 6 77 113 -26 76 49 -95 -32 83 -37 -4 -70 96 -57 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> 19-dic-2007 15:15:18
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
>
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost.local
> domain:4002,localhost.localdomain,4002, alive=5097079,id={73 -17 6 77 113
> -26 76 49 -95 -32 83 -37 -4 -70 96 -57 }, payload={}, command={},
domain={},
> ]]
> 19-dic-2007 15:15:18 org.apache.catalina.ha.tcp.SimpleTcpCluster send
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
> ms.).; Faulty members:tcp://localhost.localdomain:4002;
>         at
>
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
> lelNioSender.java:97)
>         at
>
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
> oledParallelSender.java:53)
>         at
>
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
> icationTransmitter.java:80)
>         at
>
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
> inator.java:78)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
>
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
> age(ThroughputInterceptor.java:61)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
>
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
> dMessage(MessageDispatchInterceptor.java:73)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
>
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
> (TcpFailureDetector.java:87)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at
>
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
>         at
>
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
> er.java:814)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
> a:535)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
> plicationValve.java:517)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
> onValve.java:428)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
> )
>         at
>
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
>         at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
>         at
>
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
> 11Protocol.java:584)
>         at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>         at java.lang.Thread.run(Thread.java:619)
> ================
>
> INSTANCE 2
> ================
> 19-dic-2007 14:23:15
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:20071 messages
>         Sent:11,93 MB (total)
>         Sent:11,93 MB (application)
>         Time:36,89 seconds
>         Tx Speed:0,32 MB/sec (total)
>         TxSpeed:0,32 MB/sec (application)
>         Error Msg:0
>         Rx Msg:20000 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:11,90 MB]
>
> 19-dic-2007 14:30:25
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
>
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://loc
> alhost.localdomain:4001,localhost.localdomain,4001, alive=2407419,id={-51
> -26 127 90 42 126 76 5 -98 -54 48 -28 7 -91 106 127 }, payload={},
> command={}, domain={}, ]] message. Will verify.
> 19-dic-2007 14:30:25
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
>
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://localhost.local
> domain:4001,localhost.localdomain,4001, alive=2407419,id={-51 -26 127 90
42
> 126 76 5 -98 -54 48 -28 7 -91 106 127 }, payload={}, command={},
domain={},
> ]]
> 19-dic-2007 14:30:25 org.apache.catalina.ha.tcp.SimpleTcpCluster send
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
> ms.).; Faulty members:tcp://localhost.localdomain:4001;
>         at
>
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
> lelNioSender.java:97)
>         at
>
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
> oledParallelSender.java:53)
>         at
>
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
> icationTransmitter.java:80)
>         at
>
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
> inator.java:78)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
>
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
> age(ThroughputInterceptor.java:61)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
>
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
> dMessage(MessageDispatchInterceptor.java:73)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
>
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
> (TcpFailureDetector.java:87)
>         at
>
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
> nterceptorBase.java:75)
>         at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>         at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>         at
>
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
>         at
>
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpClust
> er.java:814)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.jav
> a:535)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(Re
> plicationValve.java:517)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(Replicati
> onValve.java:428)
>         at
>
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362
> )
>         at
>
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
>         at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
>         at
>
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
> 11Protocol.java:584)
>         at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>         at java.lang.Thread.run(Thread.java:619)
> 19-dic-2007 14:37:54
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:30000 messages
>         Sent:17,81 MB (total)
>         Sent:17,82 MB (application)
>         Time:53,83 seconds
>         Tx Speed:0,33 MB/sec (total)
>         TxSpeed:0,33 MB/sec (application)
>         Error Msg:1
>         Rx Msg:29905 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:17,75 MB]
> 19-dic-2007 14:38:04
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:30108 messages
>         Sent:17,88 MB (total)
>         Sent:17,88 MB (application)
>         Time:53,96 seconds
>         Tx Speed:0,33 MB/sec (total)
>         TxSpeed:0,33 MB/sec (application)
>         Error Msg:1
>         Rx Msg:30000 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:17,81 MB]
> 19-dic-2007 14:52:22
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:40000 messages
>         Sent:23,72 MB (total)
>         Sent:23,72 MB (application)
>         Time:70,65 seconds
>         Tx Speed:0,34 MB/sec (total)
>         TxSpeed:0,34 MB/sec (application)
>         Error Msg:1
>         Rx Msg:39746 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:23,57 MB]
> 19-dic-2007 14:54:09
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
> INFO: ThroughputInterceptor Report[
>         Tx Msg:40204 messages
>         Sent:23,84 MB (total)
>         Sent:23,84 MB (application)
>         Time:70,92 seconds
>         Tx Speed:0,34 MB/sec (total)
>         TxSpeed:0,34 MB/sec (application)
>         Error Msg:1
>         Rx Msg:40000 messages
>         Rx Speed:0,01 MB/sec (since 1st msg)
>         Received:23,71 MB]
> =============
>
> The server is working, both instances, no hang or delay detected.
>
> What is this error talking about? It is really important the fact of a
> disappeared member?
>
> Thanks for your time
> Regards
> Raúl.
>
> -----Mensaje original-----
> De: Peter Rossbach [mailto:pr@objektpark.de] 
> Enviado el: miércoles, 19 de diciembre de 2007 13:14
> Para: Tomcat Developers List
> Asunto: Re: Tomcat 6 - Cluster error.
>
> I think your Membership DropTime 4sec is very small. Some more Load  
> or a FULL GC can drop your member.
> Nomally I use 30 sec.
>
> Peter
>
> Am 19.12.2007 um 11:40 schrieb Raúl García:
>
>   
>> Hi,
>>
>> We are using tomcat 6.0.14.
>>
>> We use a cluster working at the same machine,
>> we start it and it seems to be ok, replication is working fine.
>>
>> But within the next 12h from startup, we see this error at the  
>> catalina.out
>> log file, of the first instance, the error repeats forever until we  
>> stop
>> both
>>
>> instances and restart them again.
>>
>> The server is supporting a 0.5 hit per seccond aprox.
>>
>> CATALINA.OUT
>> ===============
>> Dec 19, 2007 10:07:30 AM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> performBasicCheck
>> WARNING: Member added, even though we werent
>> notified:org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>> localhost:40
>> 02,localhost,4002,
>>
>> alive=165023279,id={-42 -48 112 17 -57 -2 73 -111 -109 113 -93 84 6  
>> 91 -72
>> 102 }, payload={}, command={}, domain={}, ]
>> Dec 19, 2007 10:07:30 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
>> memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>> localhost:4002,
>> localhost,4002, alive=165023279,id={-42 -48 112 17 -57
>>
>> -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={}, command={},
>> domain={}, ]
>> Dec 19, 2007 10:07:34 AM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl 
>> [tcp://loc
>> alhost:4002,localhost,4002, alive=165028289,id={-42 -48 112 17 -57
>>
>> -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={}, command={},
>> domain={}, ]] message. Will verify.
>> Dec 19, 2007 10:07:34 AM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still
>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>> localhost:4002,
>> localhost,4002, alive=165028289,id={-42
>>
>> -48 112 17 -57 -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={},
>> command={}, domain={}, ]]
>> Dec 19, 2007 10:07:34 AM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster send
>> SEVERE: Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: Operation has timed out 
>> (3000
>> ms.).; Faulty members:tcp://localhost:4002;
>>         at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage 
>> (Paral
>> lelNioSender.java:97)
>>         at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMess 
>> age(Po
>> oledParallelSender.java:53)
>>         at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessag 
>> e(Repl
>> icationTransmitter.java:80)
>>         at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage 
>> (ChannelCoord
>> inator.java:78)
>>         at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelI
>> nterceptorBase.java:75)
>>         at
>> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.se 
>> ndMess
>> age(ThroughputInterceptor.java:61)
>>         at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelI
>> nterceptorBase.java:75)
>>         at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchIntercept 
>> or.sen
>> dMessage(MessageDispatchInterceptor.java:73)
>>         at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelI
>> nterceptorBase.java:75)
>>         at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendM 
>> essage
>> (TcpFailureDetector.java:87)
>>         at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage 
>> (ChannelI
>> nterceptorBase.java:75)
>>         at
>> org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:216)
>>         at
>> org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.send 
>> (SimpleTcpCluster.java:835)
>>         at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain 
>> (SimpleTcpClust
>> er.java:814)
>>         at
>> org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java: 
>> 586)
>>         at
>> org.apache.catalina.ha.session.DeltaManager.sendCreateSession 
>> (DeltaManager.j
>> ava:575)
>>         at
>> org.apache.catalina.ha.session.DeltaManager.createSession 
>> (DeltaManager.java:
>> 551)
>>         at
>> org.apache.catalina.ha.session.DeltaManager.createSession 
>> (DeltaManager.java:
>> 534)
>>         at
>> org.apache.catalina.connector.Request.doGetSession(Request.java:2312)
>>         at
>> org.apache.catalina.connector.Request.getSession(Request.java:2075)
>>         at
>> org.apache.catalina.connector.RequestFacade.getSession 
>> (RequestFacade.java:83
>> 3)
>>         at pad.kernel.Resolver.service(Resolver.java:266)
>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java: 
>> 803)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter 
>> (Application
>> FilterChain.java:290)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter 
>> (ApplicationFilterCh
>> ain.java:206)
>>         at pad.kernel.EntryPointFilter.doFilter 
>> (EntryPointFilter.java:365)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter 
>> (Application
>> FilterChain.java:235)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter 
>> (ApplicationFilterCh
>> ain.java:206)
>>         at
>> org.apache.catalina.core.StandardWrapperValve.invoke 
>> (StandardWrapperValve.ja
>> va:219)
>>         at
>> org.apache.catalina.core.StandardContextValve.invoke 
>> (StandardContextValve.ja
>> va:175)
>>         at
>> org.apache.catalina.core.StandardHostValve.invoke 
>> (StandardHostValve.java:128
>> )
>>         at
>> org.apache.catalina.valves.ErrorReportValve.invoke 
>> (ErrorReportValve.java:102
>> )
>>         at
>> org.apache.catalina.valves.RequestFilterValve.process 
>> (RequestFilterValve.jav
>> a:269)
>>         at
>> org.apache.catalina.valves.RemoteAddrValve.invoke 
>> (RemoteAddrValve.java:81)
>>         at
>> org.apache.catalina.core.StandardEngineValve.invoke 
>> (StandardEngineValve.java
>> :109)
>>         at
>> org.apache.catalina.ha.tcp.ReplicationValve.invoke 
>> (ReplicationValve.java:347
>> )
>>         at
>> org.apache.catalina.connector.CoyoteAdapter.service 
>> (CoyoteAdapter.java:263)
>>         at
>> org.apache.coyote.http11.Http11Processor.process 
>> (Http11Processor.java:844)
>>         at
>> org.apache.coyote.http11.Http11Protocol 
>> $Http11ConnectionHandler.process(Http
>> 11Protocol.java:584)
>>         at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java: 
>> 447)
>>         at java.lang.Thread.run(Thread.java:619)
>> Dec 19, 2007 10:07:36 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
>> memberDisappeared
>> INFO: Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>> localhost
>> :4002,localhost,4002, alive=165029291,id={-42 -48 112 17
>>
>> -57 -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={}, command={},
>> domain={}, ]
>> Dec 19, 2007 10:07:36 AM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> performBasicCheck
>> INFO: Suspect member, confirmed
>> dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>> localhost:4002,
>> localhost,4002, alive=165029291,id={-42 -48 112
>>
>> 17 -57 -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={},  
>> command={},
>> domain={}, ]]
>> Dec 19, 2007 10:07:39 AM  
>> org.apache.catalina.ha.tcp.SimpleTcpCluster send
>> SEVERE: Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: No destination given;  
>> No faulty
>> members identified.
>>         at
>> org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:194)
>>         at
>> org.apache.catalina.tribes.group.GroupChannel.send 
>> (GroupChannel.java:175)
>>         at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.send 
>> (SimpleTcpCluster.java:835)
>>         at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain 
>> (SimpleTcpClust
>> er.java:814)
>>         at
>> org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java: 
>> 586)
>>         at
>> org.apache.catalina.ha.session.DeltaManager.sessionExpired 
>> (DeltaManager.java
>> :1211)
>>         at
>> org.apache.catalina.ha.session.DeltaSession.expire 
>> (DeltaSession.java:366)
>>         at
>> org.apache.catalina.ha.session.DeltaSession.expire 
>> (DeltaSession.java:352)
>>         at
>> org.apache.catalina.session.StandardSession.expire 
>> (StandardSession.java:660)
>>         at
>> org.apache.catalina.session.StandardSession.invalidate 
>> (StandardSession.java:
>> 1111)
>>         at
>> org.apache.catalina.session.StandardSessionFacade.invalidate 
>> (StandardSession
>> Facade.java:150)
>>         at pad.kernel.Resolver.service(Resolver.java:623)
>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java: 
>> 803)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter 
>> (Application
>> FilterChain.java:290)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter 
>> (ApplicationFilterCh
>> ain.java:206)
>>         at pad.kernel.EntryPointFilter.doFilter 
>> (EntryPointFilter.java:365)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter 
>> (Application
>> FilterChain.java:235)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter 
>> (ApplicationFilterCh
>> ain.java:206)
>>         at
>> org.apache.catalina.core.StandardWrapperValve.invoke 
>> (StandardWrapperValve.ja
>> va:219)
>>         at
>> org.apache.catalina.core.StandardContextValve.invoke 
>> (StandardContextValve.ja
>> va:175)
>>         at
>> org.apache.catalina.core.StandardHostValve.invoke 
>> (StandardHostValve.java:128
>> )
>>         at
>> org.apache.catalina.valves.ErrorReportValve.invoke 
>> (ErrorReportValve.java:102
>> )
>>         at
>> org.apache.catalina.valves.RequestFilterValve.process 
>> (RequestFilterValve.jav
>> a:269)
>>         at
>> org.apache.catalina.valves.RemoteAddrValve.invoke 
>> (RemoteAddrValve.java:81)
>>         at
>> org.apache.catalina.core.StandardEngineValve.invoke 
>> (StandardEngineValve.java
>> :109)
>>         at
>> org.apache.catalina.ha.tcp.ReplicationValve.invoke 
>> (ReplicationValve.java:347
>> )
>>         at
>> org.apache.catalina.connector.CoyoteAdapter.service 
>> (CoyoteAdapter.java:263)
>>         at
>> org.apache.coyote.http11.Http11Processor.process 
>> (Http11Processor.java:844)
>>         at
>> org.apache.coyote.http11.Http11Protocol 
>> $Http11ConnectionHandler.process(Http
>> 11Protocol.java:584)
>>         at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java: 
>> 447)
>>         at java.lang.Thread.run(Thread.java:619)
>> Dec 19, 2007 10:07:39 AM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl 
>> [tcp://loc
>> alhost:4002,localhost,4002, alive=165033299,id={-42 -48 112 17 -57
>>
>> -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={}, command={},
>> domain={}, ]] message. Will verify.
>> Dec 19, 2007 10:07:39 AM
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still
>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp:// 
>> localhost:4002,
>> localhost,4002, alive=165033299,id={-42
>>
>> -48 112 17 -57 -2 73 -111 -109 113 -93 84 6 91 -72 102 }, payload={},
>> command={}, domain={}, ]]
>>
>> [...]
>>
>> And repeats again, and again...
>>
>> ========================
>>
>>
>> SERVER.XML (Instance_1)
>> =======================
>> [...] Connector
>>
>>     <Connector port="8081" protocol="HTTP/1.1"  
>> maxHttpHeaderSize="8192"
>> emptySessionPath="true"
>>                maxThreads="150" minSpareThreads="100"  
>> maxSpareThreads="300"
>>                enableLookups="false" redirectPort="81443"  
>> acceptCount="1000"
>>                debug="0" connectionTimeout="20000"
>> disableUploadTimeout="true"
>>                compression="on"
>> 			   compressionMinSize="2048"
>> 			   noCompressionUserAgents="gozilla, traviata"
>> 			   compressableMimeType="text/html,text/xml" />
>>
>>
>> [...] Cluster Tag
>>
>> 	<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>                  channelSendOptions="6">
>>
>>
>>           <Manager  
>> className="org.apache.catalina.ha.session.DeltaManager"
>>                    expireSessionsOnShutdown="false"
>>                    notifyListenersOnReplication="true"/>
>>
>>           <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>>             <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>>                         address="228.0.0.8"
>>                         port="45578"
>>                         frequency="1000"
>>                         dropTime="4000"/>
>>             <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>                       address="localhost"
>>                       port="4001"
>>                       autoBind="100"
>>                       selectorTimeout="100"
>>                       maxThreads="12"/>
>>
>>             <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter 
>> ">
>>               <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSend 
>> er"/>
>>             </Sender>
>>             <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDet 
>> ector"
>> />
>>             <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispat 
>> ch15In
>> terceptor"/>
>>             <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInt 
>> ercept
>> or"/>
>>           </Channel>
>>
>>           <Valve  
>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>
>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.* 
>> \.txt;"/>
>>
>>           <Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>                     tempDir="/tmp/war-temp/"
>>                     deployDir="/tmp/war-deploy/"
>>                     watchDir="/tmp/war-listen/"
>>                     watchEnabled="false"/>
>>           <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>         </Cluster>
>> [...]
>> ================
>>
>> WEB.XML Instance 1 and 2
>> ========================
>> We added the  	<Context distributable="true" /> tag
>>
>>
>> SERVER.XML (Instance_2)
>> =======================
>> [...] Connector Tag
>>
>>     <Connector port="8082" protocol="HTTP/1.1"  
>> maxHttpHeaderSize="8192"
>> emptySessionPath="true"
>>                maxThreads="150" minSpareThreads="100"  
>> maxSpareThreads="300"
>>                enableLookups="false" redirectPort="82443"  
>> acceptCount="1000"
>>                debug="0" connectionTimeout="20000"
>> disableUploadTimeout="true"
>>                compression="on"
>> 			   compressionMinSize="2048"
>> 			   noCompressionUserAgents="gozilla, traviata"
>> 			   compressableMimeType="text/html,text/xml" />
>>
>> [...] Cluster Tag
>>
>> 	<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>                  channelSendOptions="6">
>>
>>           <Manager  
>> className="org.apache.catalina.ha.session.DeltaManager"
>>                    expireSessionsOnShutdown="false"
>>                    notifyListenersOnReplication="true"/>
>>
>>           <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>>             <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>>                         address="228.0.0.8"
>>                         port="45578"
>>                         frequency="1000"
>>                         dropTime="4000"/>
>>             <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>                       address="localhost"
>>                       port="4002"
>>                       autoBind="100"
>>                       selectorTimeout="100"
>>                       maxThreads="12"/>
>>
>>             <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter 
>> ">
>>               <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSend 
>> er"/>
>>             </Sender>
>>             <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDet 
>> ector"
>> />
>>             <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispat 
>> ch15In
>> terceptor"/>
>>             <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInt 
>> ercept
>> or"/>
>>           </Channel>
>>           <Valve  
>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>
>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.* 
>> \.txt;"/>
>>           <Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>                     tempDir="/tmp/war-temp/"
>>                     deployDir="/tmp/war-deploy/"
>>                     watchDir="/tmp/war-listen/"
>>                     watchEnabled="false"/>
>>           <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>         </Cluster>
>> [...]
>> ================
>>
>>
>> Can someone help us with this problem?
>> Any suggestion, idea?
>>
>> I can send more config files or log text if needed.
>>
>> Regards
>> Raúl.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>>     
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Mime
View raw message