From users-return-265574-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Thu Sep 20 16:29:25 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 61468180671 for ; Thu, 20 Sep 2018 16:29:24 +0200 (CEST) Received: (qmail 96685 invoked by uid 500); 20 Sep 2018 14:29:22 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 96674 invoked by uid 99); 20 Sep 2018 14:29:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Sep 2018 14:29:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0E17EC546C for ; Thu, 20 Sep 2018 14:29:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.909 X-Spam-Level: X-Spam-Status: No, score=0.909 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIM_INVALID=0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=neutral reason="invalid (public key: not available)" header.d=guilfordgroup.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id C_YbLgs9c9bS for ; Thu, 20 Sep 2018 14:29:20 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 214FE5F419 for ; Thu, 20 Sep 2018 14:29:20 +0000 (UTC) Received: by mail-wm1-f53.google.com with SMTP id o18-v6so11350649wmc.0 for ; Thu, 20 Sep 2018 07:29:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=guilfordgroup.com; s=google; h=from:references:in-reply-to:mime-version:thread-index:date :message-id:subject:to; bh=dOD8N31qneiYayh4IIpl0V+4Ojiz9uklcaE77T1TNQY=; b=Y2ALRtxzRcEQ5rtVkL8CeYLJGQqMSnBoTmpbaeqX0/8CbgoyYhZoBvoDGPJ9DC5Lel /SBXuHPMZlk8p7gM/R8hs4gks54hudp1pjBMLGz7ErV284mDhAn1726OhntA10iHHQHQ NjwZtFfgGfam3HkFvTsVbbRw8nwtao802wfvk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to; bh=dOD8N31qneiYayh4IIpl0V+4Ojiz9uklcaE77T1TNQY=; b=PZt7wqX9Ea+vABOUilEiph2ryYYogqn1PzeVjFBrwM7YQqDmWPKakycggzXEJjAllx GfBKYuuy84H5TCGkWIU05KJqbCqH8YAY+YMm9TdeEhtUM0rJrmNutAcuGo8Xs+ZR+pnZ u+DapRte8yXUTiReY3/WxikygooTh/3X+ipZMcBtsTh64z71yK/3h/q/2N0lEhIjRImB hAujeLCQ8E+DN/rNWpQFqMp1fnVwCa9yg4axdMw/wYEzzNcdPLDTcp9aQ7NoB6unggQK 2XGJUniPknlVVtJWKf4929D7akFeQfiasxV648zJnhKyYJkcHclwntfX4RUSv8dt5DSu yq3g== X-Gm-Message-State: APzg51CQivTLhmw7sg79G0B3clo3+cSw1a/AVkf/NWhO2LAEieLbCs1Q mVcdTiPIbJU/6VboI7Xh47UD1ZFAAO8QB8LQwEATklLi X-Google-Smtp-Source: ANB0VdaHAoURrW6jRhDYiyv3JULnS80hlM6N8pC2Q7r8iTJ4nlK7dsMAGldTYgzH7fheE1vhZE2dj7hc2koU342MwH8= X-Received: by 2002:a1c:ee15:: with SMTP id m21-v6mr2916838wmh.112.1537453759104; Thu, 20 Sep 2018 07:29:19 -0700 (PDT) From: Scott Evans References: <118f7f3a-6d1a-6090-dca6-31002154157f@christopherschultz.net> <58252d9e-a0dd-b540-3b66-cc3696eccdc4@christopherschultz.net> 7f2129cd3d36233ebac44ae33a8ea409@mail.gmail.com In-Reply-To: 7f2129cd3d36233ebac44ae33a8ea409@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQNUn3Zb2ArjYpEW9AIlgeHCvoCwxQF9LJgRAoPMC7sCSKy/iaG5171wgAx3qEA= Date: Thu, 20 Sep 2018 10:26:36 -0400 Message-ID: <8b0277dea37a6d8b53c60e279952ad68@mail.gmail.com> Subject: RE: [OT] Tomcat Clustering Support To: Tomcat Users List Content-Type: text/plain; charset="UTF-8" -----Original Message----- From: Scott Evans Sent: Monday, September 17, 2018 10:57 AM To: 'Tomcat Users List' ; 'users@tomcat.apache.org' Subject: RE: [OT] Tomcat Clustering Support -----Original Message----- From: Christopher Schultz Sent: Tuesday, August 28, 2018 3:35 PM To: users@tomcat.apache.org Subject: Re: [OT] Tomcat Clustering Support -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Scott, I'm no Tomcat-clustering expert, but... On 8/28/18 13:59, Scott Evans wrote: > channelSendOptions="10" channelStartOptions="3"> > > expireSessionsOnShutdown="false" > notifyListenersOnReplication="true" mapSendOptions="10" /> > > className="org.apache.catalina.tribes.group.GroupChannel"> > className="org.apache.catalina.tribes.membership.McastService" > address="228.1.0.13" port="45522" frequency="500" dropTime="15000" > soTimeout="10000" domain="mercer"/> className="org.apache.catalina.tribes.transport.nio.NioReceiver" > address="10.255.250.34" port="4003" selectorTimeout="8000" > maxThreads="25" /> > > className="org.apache.catalina.tribes.transport.ReplicationTransmitter "> > > className="org.apache.catalina.tribes.transport.nio.PooledParallelSend er" > > timeout="8000" poolSize="25" > /> className="org.apache.catalina.tribes.group.interceptors.TcpPingInterc eptor"/> > > className="org.apache.catalina.tribes.group.interceptors.TcpFailureDet ector"/> > > className="org.apache.catalina.tribes.group.interceptors.MessageDispat ch15Interceptor"/> > > > > className="org.apache.catalina.tribes.group.interceptors.StaticMembers hipInterceptor"> > > className="org.apache.catalina.tribes.membership.StaticMember" > domain="mercer" uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,1,0}"/> > > className="org.apache.catalina.tribes.membership.StaticMember" > port="4001" host="10.255.250.35" domain="mercer" > uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,0,0}" /> > > className="org.apache.catalina.tribes.membership.StaticMember" > port="4002" host="10.255.250.35" domain="mercer" > uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,0,1}" /> > > className="org.apache.catalina.tribes.membership.StaticMember" > port="4004" host="10.255.250.34" domain="mercer" > uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,1,1}" /> > > > > > > > /> filter=".*javax\.faces\.resource.*"/> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> > > className="org.apache.catalina.ha.deploy.FarmWarDeployer" > tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" > watchDir="/tmp/war-listen/" watchEnabled="false"/> > > > > It looks like you have both multicast AND static membership enabled. Keichi's presentation on Clustering at ApacheCon Miami (2017) has a slide (it's slide #38 here: https://events.static.linuxfound.org/sites/events/files/slides/TomcatClu ster_3.pdf) that says that using static-membership requires that you disable multicast. Also, just confirming that you have two Tomcat nodes on one IP address (10.255.250.35, ports 4001 and 4002). Can you post a thread dump of a deadlock situation? Only the deadlocked threads should really be necessary to post. Can you replicate the deadlock without using your own full application? That is, can you create a simple application that can be used to reproduce this on a similarly-configured test instance (cluster) of Tomcat nodes? - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluFo/MACgkQHPApP6U8 pFhG4xAAkUb/Zj9HtwRle8xypc8hrmLfiifo9acIKbb1H3k/2VfYW3EjGqVRzV6c E5iGf3JFlnsDEsMIi/oSTObe/aJ15y6z1qfCpud1BRSvi1yHr8jf6W+/M4/QcMNk JerBmsx8dgoLteVq34xEld678NftgufaHpd3z5y3HnqfX0MoJkCOaYH5lUbA5MpI 61vEngWnWsLvFyTcf+h9PnkxsH5CdA0A9Hjsg56MESAyGoEZ1Jx1MkrIooFLOHVx sgxciUIosQy5wqIbpZrZMteB1T6gFSvVsoTCu2ogubJUU216xt3XEezVtksL9Kfc +1GbaDeMb65W6GlUU9W61TPb4Id/2mcQ2oUyQERctvIib7GoTcpLJFSHkKlp81GL vS3L4siQkSv1M6pIvAtnAJEVPogBgYJXnSVOObpGAmyaDkJt8k1OSCDWqVPmLfUm mIlhDGBtngxl0pEM1juLFC2ulaOGS8Vjn5VGZgXDXZVQ6xVmqBDfl9o6x+IB+KDT beOGXQKveI18K0qPjxfVtF9OyVgfeLoOzVw2AXAD8QBXorWPlEt53sbInv2r/a3l UOKGvxxGpeqmzAtEwm0GxrJsDrfJ2tTp0eIDA94n7d3tuG+zoOgOFaMxXcryieyj XXl+4+DjD7YxVAXNfUslP7eYglHh1SdJVc8/MwlH0g0fARY74/o= =eKZv -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org Hi Chris, Just getting back to this. Thanks for the info on static membership and multicast, we may give that a try. We've also finally been able to get a thread dump of a recent deadlock situation that occurred which is good news. To answer your other questions, you are correct that two Nodes are on one IP and we do not have a way to create a simple program to reproduce the issue. It is just randomly occurring, though very infrequently. For next steps, due to the sensitive nature and voluminous amount of the data contained in the thread dump, we would ask that we somehow take this offline so it's not shared on the forums. Please let me know what our options would be for that. Thanks -------------------------- Hi, just checking back in. Is there anyone who is willing to look at our thread dump on an individual basis to see what may be causing the deadlocks? We would rather not share it here since it contains sensitive information, thanks. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org