Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3C2C3200D35 for ; Tue, 7 Nov 2017 16:37:54 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3AFCB160BED; Tue, 7 Nov 2017 15:37:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 336F61609C8 for ; Tue, 7 Nov 2017 16:37:53 +0100 (CET) Received: (qmail 47871 invoked by uid 500); 7 Nov 2017 15:37:52 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 47861 invoked by uid 99); 7 Nov 2017 15:37:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Nov 2017 15:37:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 672D018EEF7 for ; Tue, 7 Nov 2017 15:37:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.379 X-Spam-Level: *** X-Spam-Status: No, score=3.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id zvo1b6fUED36 for ; Tue, 7 Nov 2017 15:37:50 +0000 (UTC) Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id BEC2B5F566 for ; Tue, 7 Nov 2017 15:37:49 +0000 (UTC) Received: by mail-lf0-f44.google.com with SMTP id a132so14984575lfa.7 for ; Tue, 07 Nov 2017 07:37:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:from:subject:date:importance:in-reply-to :references; bh=sI9CSd4wyeF2U7vNNHrvyDXeE0FUE+D+Mp5Xn+84bJc=; b=o0F4iOSoGByep/zwcKCgZbSYrFyoxCoiQwvVVdT5M4XqTIt1KWCUo7cy2uLmYdPiSj xEyzQQjXAGc4Q5M84046hXcw8YdI4O8fmLgMW9Y6SVNxZvu3nfRTR7YebdcpAmWih2RB SJdxC5BSPz4MvG983+XMjbac7KcoT5+av8BqkL/7Ct96a4rDJSt3QyHne2oGJ3K067Tq NjOC47xnJNTYlPn9oOTBLr6xQC6cxzSnBlXFlrDUFTwtBMFHZjTlHqT3r8tnTcEHH8WZ 8mNfFE/fw+2wVgHRphajnsleKrTkVosWrvkEmjoSkHhyQxTrxQYxQyq23jOCPx6JD1B+ umSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:from:subject:date :importance:in-reply-to:references; bh=sI9CSd4wyeF2U7vNNHrvyDXeE0FUE+D+Mp5Xn+84bJc=; b=GAXF1syH67DI2LwOFMuMs6MwGPba3CKWqi1v2Inr8al58Z1oSyTG2RAbUXMvV7F2iJ j89bwGc2xJmUhMBtxEYSmTDzFklwrUHf0NAoZfPWcvapGUrk1IWtfv5I5l7qSQ6OmIir ebWRkLut+QhbehQQKvXw/9PSfCoLSIyolxYSaJ4k7zUvFHYRGz+QyO+EEg7KWS+J9p4N qunPFURSmJUf04Y5Amjxg46IDDkeSuzWuYnPO9yGbnjA7W9rzF08pSyf+375AOkWBKGY EBlFfprLW7FpuPfnL2nLzVgjm5C4H5vT3xm824Sg6PITnbM4cDzp0aH5MY2LS/qLUl9T Gg0Q== X-Gm-Message-State: AMCzsaXivSWLE8pmf9bfyFWsNRT4HFwNBI632uH129iLXUOH7Trn3iZ8 Qb5gZV5ayzpMDPkyOuhVQ3ImRLg8 X-Google-Smtp-Source: ABhQp+SF6uLrQJ2oWVM7Q7tN/fZHQEh8BJPbBy8BMszHaZf/VudqQdL6I+6xCI5/7rgSLT3xAstAWg== X-Received: by 10.46.71.77 with SMTP id u74mr8100673lja.79.1510069068287; Tue, 07 Nov 2017 07:37:48 -0800 (PST) Received: from ?IPv6:::ffff:172.25.4.166? ([195.144.253.150]) by smtp.gmail.com with ESMTPSA id j71sm281019lfk.6.2017.11.07.07.37.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Nov 2017 07:37:47 -0800 (PST) Message-ID: <5a01d34b.cae9190a.d70f1.1fa3@mx.google.com> MIME-Version: 1.0 To: "user@ignite.apache.org" From: Alexey Popov Subject: RE: Node failed to startup due to deadlock Date: Tue, 7 Nov 2017 18:37:47 +0300 Importance: normal X-Priority: 3 In-Reply-To: <1509935754564-0.post@n6.nabble.com> References: <1509415438711-0.post@n6.nabble.com> <1509453850794-0.post@n6.nabble.com> <1509469595775-0.post@n6.nabble.com> <1509525447046-0.post@n6.nabble.com> <1509559564390-0.post@n6.nabble.com> <1509639745051-0.post@n6.nabble.com> <1509639977318-0.post@n6.nabble.com> <1509640988489-0.post@n6.nabble.com> <59fc1db5.c8412e0a.6f61b.6ae2@mx.google.com> <1509730199194-0.post@n6.nabble.com> <1509935754564-0.post@n6.nabble.com> Content-Type: multipart/alternative; boundary="_14B4DCA6-4151-4D9F-AFCB-DBEB31A9BCF2_" archived-at: Tue, 07 Nov 2017 15:37:54 -0000 --_14B4DCA6-4151-4D9F-AFCB-DBEB31A9BCF2_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi Naresh, I see deadlocks with=20 1. com.rover.core.dao.model.admin.AdminPreferenceTable (2 locks) 2. com.rover.core.dao.model.product.assembly.AssemblyUpi2BomTable (2 locks) 3. com.rover.core.dao.model.product.assembly.AssemblyBomUpi2PackageTable, 4. com.rover.core.dao.model.product.ProductVersionInfoTable Could you please provide a reproducible sample for this issue to move furth= er? I don=E2=80=99t have a clear understanding what can cause this issue and I = could not reproduce it. BTW, Please check your configuration, I see versions mismatch (2.3 vs 1.9) = in Ignite output: >>> Ignite ver. 2.3.0-SNAPSHOT#19691231-sha1:DEV INFO: IGNITE_HOME=3DC:\development\software\apache-ignite-fabric-1.9.0-bin Please update IGNITE_HOME to a correct path. Thank you, Alexey From: naresh.goty Sent: Monday, November 6, 2017 5:36 AM To: user@ignite.apache.org Subject: RE: Node failed to startup due to deadlock Hi Alexey, We are still seeing the deadlocks for the scenario i have specified earlier= . We tried the below two changes, but still seeing the deadlocks. Can you please provide some pointers about the issue based on the logs and threaddumps attached. 1) As Rajeev mentioned, we tried offloading event handling to application threads. 2) Applied the patch fix (https://issues.apache.org/jira/browse/IGNITE-6380#), but still the same issue. Node1.log =20 Node2.log =20 Node1_AfterNode2_Turndown.tdump =20 Node1_AfterNode2_StartedAgain.tdump =20 Node2_AfteStartedAgain.tdump =20 Regards, Naresh -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ --_14B4DCA6-4151-4D9F-AFCB-DBEB31A9BCF2_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

Hi Naresh,

<= o:p> 

I see deadlocks with

  1. com.rover.core.dao.model.admin.AdminPreferenceTable (2 = locks)
  2. com.rove= r.core.dao.model.product.assembly.AssemblyUpi2BomTable (2 locks)
  3. com.rover.core.dao.model.= product.assembly.AssemblyBomUpi2PackageTable,
  4. com.rover.core.dao.model.product.ProductVers= ionInfoTable

 

Could you please provide a reproducible sample for this issue to = move further?

 

I don=E2=80=99t have a clear understanding what can cause this issue a= nd I could not reproduce it.

 

<= p class=3DMsoNormal> 

BTW, Please c= heck your configuration, I see versions mismatch (2.3 vs 1.9) in Ignite out= put:

 

>= >> Ignite ver. 2.3.0-SNAPSHOT#19691231-sha1:DEV

INFO: IGNITE_HOME=3DC:\development\software\apache-ignite-fabric-1.9.0-b= in

 

Please= update IGNITE_HOME to a correct path.

 <= /o:p>

Thank you,
Alexey

=  

From: naresh.goty
Sent: Monday, November 6, 2017 5= :36 AM
To: user@ignite.= apache.org
Subject: RE: Node failed to startup due to deadloc= k

 

H= i Alexey,

 

We are still seeing the deadlocks for the scenario i have specified earlie= r.

We tried the below two changes, but still seeing= the deadlocks. Can you

please provide some pointer= s about the issue based on the logs and

threaddumps= attached.

 

1) As Rajeev mentioned, we tried offloading event handling to application=

threads.

2) Applied the pa= tch fix

(https://issues.apache.org/jira/browse/IGNI= TE-6380#), but still the same

issue.

 

 

Node1.log

<http://apache-igni= te-users.70518.x6.nabble.com/file/t1286/Node1.log>=C2=A0

Node2.log

<http://apache-ignite-users.= 70518.x6.nabble.com/file/t1286/Node2.log>=C2=A0

 

Node1_AfterNode2_Turndown.tdump<= /p>

<http://apache-ignite-users.70518.x6.nabble.com/= file/t1286/Node1_AfterNode2_Turndown.tdump>=C2=A0

 

Node1_AfterNode2_StartedAgain.= tdump

<http://apache-ignite-users.70518.x6.nabbl= e.com/file/t1286/Node1_AfterNode2_StartedAgain.tdump>=C2=A0

 

Node2_AfteStartedAga= in.tdump

<http://apache-ignite-users.70518.x6.na= bble.com/file/t1286/Node2_AfteStartedAgain.tdump>=C2=A0

 

 

Regards,

Naresh

 

 

 

 =

 

--

Sent from: http://apache-ignite-users.70518.x6.nabble.co= m/

 

= --_14B4DCA6-4151-4D9F-AFCB-DBEB31A9BCF2_--