From user-return-24850-archive-asf-public=cust-asf.ponee.io@ignite.apache.org  Tue Feb 19 06:03:58 2019
Return-Path: <user-return-24850-archive-asf-public=cust-asf.ponee.io@ignite.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by mx-eu-01.ponee.io (Postfix) with SMTP id 956B018060E
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 19 Feb 2019 07:03:57 +0100 (CET)
Received: (qmail 29543 invoked by uid 500); 19 Feb 2019 06:03:56 -0000
Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@ignite.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@ignite.apache.org>
List-Post: <mailto:user@ignite.apache.org>
List-Id: <user.ignite.apache.org>
Reply-To: user@ignite.apache.org
Delivered-To: mailing list user@ignite.apache.org
Received: (qmail 29532 invoked by uid 99); 19 Feb 2019 06:03:56 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Feb 2019 06:03:56 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D1344C0467
	for <user@ignite.apache.org>; Tue, 19 Feb 2019 06:03:55 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 0.972
X-Spam-Level:
X-Spam-Status: No, score=0.972 tagged_above=-999 required=6.31
	tests=[RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001,
	SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
	by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024)
	with ESMTP id PBdMdTeIw3SG for <user@ignite.apache.org>;
	Tue, 19 Feb 2019 06:03:49 +0000 (UTC)
Received: from n6.nabble.com (n6.nabble.com [162.255.23.37])
	by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 472025F1AD
	for <user@ignite.apache.org>; Tue, 19 Feb 2019 06:03:47 +0000 (UTC)
Received: from n6.nabble.com (localhost [127.0.0.1])
	by n6.nabble.com (Postfix) with ESMTP id 8F192C6BB81C
	for <user@ignite.apache.org>; Mon, 18 Feb 2019 23:03:41 -0700 (MST)
Date: Mon, 18 Feb 2019 23:03:41 -0700 (MST)
From: "garima.j" <garima.j@ril.com>
To: user@ignite.apache.org
Message-ID: <1550556221543-0.post@n6.nabble.com>
Subject: Ignite Node crashed in middle of checkpoint and data loss
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hello,

We have an ignite cluster of 3 nodes (16GB RAM, 50GB disk space each)and
have given 10GB (off-heap) to data region and (Xms) 2GB and (Xmx) 3GB to the
nodes. 

One node went down and while restarting the node, I get the exception that
Ignite node crashed in the middle of checkpoint and JVM crash after that. 

Ignite configuration : 
 <property name="transactionConfiguration">
            <bean
class="org.apache.ignite.configuration.TransactionConfiguration">
                
                <property name="TxTimeoutOnPartitionMapExchange"
value="20000"/>
                
            </bean>
        </property>
        
        <property name="publicThreadPoolSize" value="64"/>
        <property name="systemThreadPoolSize" value="32"/>

Data Storage configuration : 
<property name="dataStorageConfiguration">
            <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="defaultDataRegionConfiguration">
                    <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="metricsEnabled" value="true"/>
                        <property name="initialSize" value="#{4L * 1024 *
1024 * 1024}"/>
                        <property name="maxSize" value="#{10L * 1024 * 1024
* 1024}"/>
                        <property name="checkpointPageBufferSize"
value="#{1L * 1024 * 1024 * 1024}"/>
                        <property name="pageEvictionMode"
value="RANDOM_2_LRU"/>
                    </bean>
                </property>
                <property name="storagePath" value="/data1/data/datastore"/>
                <property name="walPath" value="/data2/data/wal"/>
                <property name="walArchivePath"
value="/data2/data/wal/archive"/>
                <property name="walMode" value="LOG_ONLY"/>
                <property name="pageSize" value="#{4 * 1024}"/>
                <property name="writeThrottlingEnabled" value="true"/>
                <property name="walHistorySize" value="1"/>
                <property name="metricsEnabled" value="true"/>
                <property name="checkpointFrequency" value="60000"/>
            </bean>
        </property>

Cache configuration : 
<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="XXX"/>
                    <property name="sqlSchema" value="PUBLIC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="queryParallelism" value="8"/>
                    <property name="backups" value="1"/>
                    <property name="rebalanceThrottle" value="100"/>
                    <property name="rebalanceThreadPoolSize" value="4"/>
                    <property name="atomicityMode"
value="TRANSACTIONAL_SNAPSHOT"/>

Please find the logs(FINE level) and JVM crash logs.

hs_err_pid6456.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2241/hs_err_pid6456.log>  
ignite-8393e373.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2241/ignite-8393e373.log>  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/