Return-Path: X-Original-To: apmail-geode-issues-archive@minotaur.apache.org Delivered-To: apmail-geode-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9C07818F97 for ; Mon, 16 Nov 2015 19:47:22 +0000 (UTC) Received: (qmail 68904 invoked by uid 500); 16 Nov 2015 19:47:22 -0000 Delivered-To: apmail-geode-issues-archive@geode.apache.org Received: (qmail 68874 invoked by uid 500); 16 Nov 2015 19:47:22 -0000 Mailing-List: contact issues-help@geode.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.incubator.apache.org Delivered-To: mailing list issues@geode.incubator.apache.org Received: (qmail 68865 invoked by uid 99); 16 Nov 2015 19:47:22 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2015 19:47:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E13DB180A18 for ; Mon, 16 Nov 2015 19:47:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.771 X-Spam-Level: * X-Spam-Status: No, score=1.771 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id iz9TETomGShb for ; Mon, 16 Nov 2015 19:47:12 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with SMTP id AF770441A4 for ; Mon, 16 Nov 2015 19:47:11 +0000 (UTC) Received: (qmail 66819 invoked by uid 99); 16 Nov 2015 19:47:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2015 19:47:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 20E602C1F6B for ; Mon, 16 Nov 2015 19:47:11 +0000 (UTC) Date: Mon, 16 Nov 2015 19:47:11 +0000 (UTC) From: "Barry Oglesby (JIRA)" To: issues@geode.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (GEODE-514) DistributionManagerDUnitTest.testKickOutSickMember has suspect string MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/GEODE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007176#comment-15007176 ] Barry Oglesby edited comment on GEODE-514 at 11/16/15 7:47 PM: --------------------------------------------------------------- This failure reproduced in: Geode_develop_DistributedTests Build #594 (Nov 16, 2015 5:09:12 AM). Revision: 88da702593157d8a0c014295cab16149fc088dfc https://brazil.gemstone.com:8080/job/Geode_develop_DistributedTests/594/testReport/junit/com.gemstone.gemfire.distributed.internal/DistributionManagerDUnitTest/testKickOutSickMember/ {noformat} Suspicious strings were written to the log during this run. Fix the strings or use DistributedTestCase.addExpectedException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 981 [fatal 2015/11/16 08:30:55.682 PST tid=0x7b0] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856):10709, timor(5801):50799] ----------------------------------------------------------------------- Found suspect string in log4j at line 985 [fatal 2015/11/16 08:30:55.683 PST tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856):10709, timor(5801):50799] {noformat} Here is output from the log file: {noformat} [setup] START TEST DistributionManagerDUnitTest.testKickOutSickMember [locator][warn 2015/11/16 08:30:10.209 PST tid=0x39] failed to collect all ACKs (2) for view preparation [timor(5835:locator):26739|173] [timor(5835:locator):26739/53740, timor(5801):50799/3528] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:10.209 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:10.210 PST tid=0x2f5] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:10.211 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|174] [timor(5835:locator):26739/53740] crashed mbrs: [timor(5801):50799/3528]] (1 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:10.211 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|174] [timor(5835:locator):26739/53740] crashed mbrs: [timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:10.211 PST tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10 [locator] [locator][info 2015/11/16 08:30:10.365 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|175] [timor(5835:locator):26739/53740, timor(5801):50799/3528, timor(putter:5801):55014/18309]] (3 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:11.812 PST tid=0x2f9] Admitting member :55014>. Now there are 2 non-admin member(s). [locator] [locator][warn 2015/11/16 08:30:22.805 PST tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator):26739|175] [timor(5835:locator):26739/53740, timor(5801):50799/3528, timor(putter:5801):55014/18309] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:22.805 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:22.805 PST tid=0x2fc] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:22.806 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|176] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(5801):50799/3528]] (2 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:22.806 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|176] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:22.806 PST tid=0x24] old membership weight=3, loss threshold=2 and failed weight=10 [locator] [locator][info 2015/11/16 08:30:22.807 PST tid=0x24] Membership: lead member is now timor(putter:5801):55014 [locator] [locator][info 2015/11/16 08:30:22.808 PST tid=0x2fe] GemFire failure detection is now monitoring timor(putter:5801):55014 [locator] [locator][info 2015/11/16 08:30:23.827 PST tid=0x102] Member timor(putter:5801):55014 is equivalent or in the same redundancy zone. [locator] [locator][info 2015/11/16 08:30:24.017 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|177] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528, timor(sleeper:5856):10709/55069]] (4 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:24.023 PST tid=0x300] Admitting member :10709>. Now there are 3 non-admin member(s). [locator] [locator][warn 2015/11/16 08:30:36.458 PST tid=0x39] failed to collect all ACKs (4) for view preparation [timor(5835:locator):26739|177] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528, timor(sleeper:5856):10709/55069] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:36.458 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:36.458 PST tid=0x302] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:36.459 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|178] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(sleeper:5856):10709/55069] crashed mbrs: [timor(5801):50799/3528]] (3 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:36.460 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|178] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(sleeper:5856):10709/55069] crashed mbrs: [timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:36.460 PST tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10 [locator] [locator][info 2015/11/16 08:30:38.044 PST tid=0x102] Member timor(sleeper:5856):10709 is equivalent or in the same redundancy zone. [locator] [locator][info 2015/11/16 08:30:38.084 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(sleeper:5856):10709. [locator] [locator][info 2015/11/16 08:30:43.080 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(putter:5801):55014. Reason=Failed to respond within ack-wait-threshold [locator] [locator][info 2015/11/16 08:30:43.085 PST tid=0x305] timor(5835:locator):26739: No suspect verification response received from timor(sleeper:5856):10709 in 5001 milliseconds: I believe it is gone. [locator] [locator][info 2015/11/16 08:30:43.238 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|179] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528] crashed mbrs: [timor(sleeper:5856):10709/55069]] (3 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:48.082 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(putter:5801):55014. Reason=still being suspected [locator] [locator][info 2015/11/16 08:30:53.085 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(putter:5801):55014. Reason=still being suspected [locator] [locator][info 2015/11/16 08:30:53.085 PST tid=0x305] timor(5835:locator):26739: No suspect verification response received from timor(sleeper:5856):10709 in 5002 milliseconds: I believe it is gone. [locator] [locator][warn 2015/11/16 08:30:55.679 PST tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator):26739|179] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528] crashed mbrs: [timor(sleeper:5856):10709/55069] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:55.679 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:55.679 PST tid=0x307] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:55.680 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|180] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(sleeper:5856):10709/55069, timor(5801):50799/3528]] (2 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:55.680 PST tid=0x39] [locator] member timor(5835:locator):26739 has a weight of 3 [locator] member timor(putter:5801):55014 has a weight of 15 [locator] member timor(sleeper:5856):10709 has a weight of 10 [locator] [locator][info 2015/11/16 08:30:55.680 PST tid=0x39] quorum weight calculation: oldWeight=28 failureWeight=10 threshold=51% (failure weight must be < 14) [locator] [locator][info 2015/11/16 08:30:55.681 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|180] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(sleeper:5856):10709/55069, timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:55.681 PST tid=0x24] old membership weight=28, loss threshold=14 and failed weight=20 [locator] [locator][info 2015/11/16 08:30:55.682 PST tid=0x3c] Member at timor(sleeper:5856):10709 unexpectedly left the distributed cache: departed JGroups view [locator] [locator][fatal 2015/11/16 08:30:55.683 PST tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856):10709, timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:55.691 PST tid=0x102] Member at timor(putter:5801):55014 gracefully left the distributed cache: shutdown message received [locator] [locator][info 2015/11/16 08:30:55.835 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|181] [timor(5835:locator):26739/53740, timor(5801):50799/3528]] (2 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:55.836 PST tid=0x2fe] Member timor(putter:5801):55014 shut down with normal termination. [locator] [locator][info 2015/11/16 08:30:55.853 PST tid=0x12] Received method: dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221 [locator] [locator][info 2015/11/16 08:30:55.853 PST tid=0x12] Got result: null [locator] from dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221 (took 0 ms) [locator] {noformat} was (Author: barry.oglesby): This failure reproduced in Geode_develop_DistributedTests Build #594 (Nov 16, 2015 5:09:12 AM). https://brazil.gemstone.com:8080/job/Geode_develop_DistributedTests/594/testReport/junit/com.gemstone.gemfire.distributed.internal/DistributionManagerDUnitTest/testKickOutSickMember/ {noformat} Suspicious strings were written to the log during this run. Fix the strings or use DistributedTestCase.addExpectedException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 981 [fatal 2015/11/16 08:30:55.682 PST tid=0x7b0] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856):10709, timor(5801):50799] ----------------------------------------------------------------------- Found suspect string in log4j at line 985 [fatal 2015/11/16 08:30:55.683 PST tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856):10709, timor(5801):50799] {noformat} Here is output from the log file: {noformat} [setup] START TEST DistributionManagerDUnitTest.testKickOutSickMember [locator][warn 2015/11/16 08:30:10.209 PST tid=0x39] failed to collect all ACKs (2) for view preparation [timor(5835:locator):26739|173] [timor(5835:locator):26739/53740, timor(5801):50799/3528] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:10.209 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:10.210 PST tid=0x2f5] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:10.211 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|174] [timor(5835:locator):26739/53740] crashed mbrs: [timor(5801):50799/3528]] (1 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:10.211 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|174] [timor(5835:locator):26739/53740] crashed mbrs: [timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:10.211 PST tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10 [locator] [locator][info 2015/11/16 08:30:10.365 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|175] [timor(5835:locator):26739/53740, timor(5801):50799/3528, timor(putter:5801):55014/18309]] (3 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:11.812 PST tid=0x2f9] Admitting member :55014>. Now there are 2 non-admin member(s). [locator] [locator][warn 2015/11/16 08:30:22.805 PST tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator):26739|175] [timor(5835:locator):26739/53740, timor(5801):50799/3528, timor(putter:5801):55014/18309] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:22.805 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:22.805 PST tid=0x2fc] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:22.806 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|176] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(5801):50799/3528]] (2 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:22.806 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|176] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:22.806 PST tid=0x24] old membership weight=3, loss threshold=2 and failed weight=10 [locator] [locator][info 2015/11/16 08:30:22.807 PST tid=0x24] Membership: lead member is now timor(putter:5801):55014 [locator] [locator][info 2015/11/16 08:30:22.808 PST tid=0x2fe] GemFire failure detection is now monitoring timor(putter:5801):55014 [locator] [locator][info 2015/11/16 08:30:23.827 PST tid=0x102] Member timor(putter:5801):55014 is equivalent or in the same redundancy zone. [locator] [locator][info 2015/11/16 08:30:24.017 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|177] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528, timor(sleeper:5856):10709/55069]] (4 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:24.023 PST tid=0x300] Admitting member :10709>. Now there are 3 non-admin member(s). [locator] [locator][warn 2015/11/16 08:30:36.458 PST tid=0x39] failed to collect all ACKs (4) for view preparation [timor(5835:locator):26739|177] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528, timor(sleeper:5856):10709/55069] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:36.458 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:36.458 PST tid=0x302] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:36.459 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|178] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(sleeper:5856):10709/55069] crashed mbrs: [timor(5801):50799/3528]] (3 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:36.460 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|178] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(sleeper:5856):10709/55069] crashed mbrs: [timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:36.460 PST tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10 [locator] [locator][info 2015/11/16 08:30:38.044 PST tid=0x102] Member timor(sleeper:5856):10709 is equivalent or in the same redundancy zone. [locator] [locator][info 2015/11/16 08:30:38.084 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(sleeper:5856):10709. [locator] [locator][info 2015/11/16 08:30:43.080 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(putter:5801):55014. Reason=Failed to respond within ack-wait-threshold [locator] [locator][info 2015/11/16 08:30:43.085 PST tid=0x305] timor(5835:locator):26739: No suspect verification response received from timor(sleeper:5856):10709 in 5001 milliseconds: I believe it is gone. [locator] [locator][info 2015/11/16 08:30:43.238 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|179] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528] crashed mbrs: [timor(sleeper:5856):10709/55069]] (3 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:48.082 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(putter:5801):55014. Reason=still being suspected [locator] [locator][info 2015/11/16 08:30:53.085 PST tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856):10709] from timor(putter:5801):55014. Reason=still being suspected [locator] [locator][info 2015/11/16 08:30:53.085 PST tid=0x305] timor(5835:locator):26739: No suspect verification response received from timor(sleeper:5856):10709 in 5002 milliseconds: I believe it is gone. [locator] [locator][warn 2015/11/16 08:30:55.679 PST tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator):26739|179] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309, timor(5801):50799/3528] crashed mbrs: [timor(sleeper:5856):10709/55069] after 12,437ms, missing ACKs from [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:55.679 PST tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:55.679 PST tid=0x307] Checking member timor(5801):50799 [locator] [locator][info 2015/11/16 08:30:55.680 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|180] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(sleeper:5856):10709/55069, timor(5801):50799/3528]] (2 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:55.680 PST tid=0x39] [locator] member timor(5835:locator):26739 has a weight of 3 [locator] member timor(putter:5801):55014 has a weight of 15 [locator] member timor(sleeper:5856):10709 has a weight of 10 [locator] [locator][info 2015/11/16 08:30:55.680 PST tid=0x39] quorum weight calculation: oldWeight=28 failureWeight=10 threshold=51% (failure weight must be < 14) [locator] [locator][info 2015/11/16 08:30:55.681 PST tid=0x24] Membership: received new view [timor(5835:locator):26739|180] [timor(5835:locator):26739/53740, timor(putter:5801):55014/18309] crashed mbrs: [timor(sleeper:5856):10709/55069, timor(5801):50799/3528] [locator] [locator][info 2015/11/16 08:30:55.681 PST tid=0x24] old membership weight=28, loss threshold=14 and failed weight=20 [locator] [locator][info 2015/11/16 08:30:55.682 PST tid=0x3c] Member at timor(sleeper:5856):10709 unexpectedly left the distributed cache: departed JGroups view [locator] [locator][fatal 2015/11/16 08:30:55.683 PST tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856):10709, timor(5801):50799] [locator] [locator][info 2015/11/16 08:30:55.691 PST tid=0x102] Member at timor(putter:5801):55014 gracefully left the distributed cache: shutdown message received [locator] [locator][info 2015/11/16 08:30:55.835 PST tid=0x39] Membership: sending new view [[timor(5835:locator):26739|181] [timor(5835:locator):26739/53740, timor(5801):50799/3528]] (2 mbrs) [locator] [locator] [locator][info 2015/11/16 08:30:55.836 PST tid=0x2fe] Member timor(putter:5801):55014 shut down with normal termination. [locator] [locator][info 2015/11/16 08:30:55.853 PST tid=0x12] Received method: dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221 [locator] [locator][info 2015/11/16 08:30:55.853 PST tid=0x12] Got result: null [locator] from dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221 (took 0 ms) [locator] {noformat} > DistributionManagerDUnitTest.testKickOutSickMember has suspect string > --------------------------------------------------------------------- > > Key: GEODE-514 > URL: https://issues.apache.org/jira/browse/GEODE-514 > Project: Geode > Issue Type: Bug > Reporter: xiaojian zhou > Labels: CI > > revision 0cc9d895b9f4465138d0fa223b0a0cadc1107893 > {noformat} > java.lang.AssertionError: Suspicious strings were written to the log during this run. > Fix the strings or use DistributedTestCase.addExpectedException to ignore. > ----------------------------------------------------------------------- > Found suspect string in log4j at line 990 > [fatal 2015/10/29 06:28:31.322 PDT tid=0xada] Possible loss of quorum due to the loss of 2 cache processes: [cc8-rh64(16364):1121, cc8-rh64(sleeper:16419):30890] > ----------------------------------------------------------------------- > Found suspect string in log4j at line 992 > [fatal 2015/10/29 06:28:31.323 PDT tid=0xe04] Possible loss of quorum due to the loss of 2 cache processes: [cc8-rh64(16364):1121, cc8-rh64(sleeper:16419):30890] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)