This only happen when I shut down the nodes with this persist-able  cache. 

After bring those nodes back, either throw this exception, or the node hang up forever  waiting the initial partition  map.

[WARN ] 2018-01-02 13:19:08.318 [main] [ig] GridCachePartitionExchangeManager - Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bfb39f71-fa1d-4547-9b49-b59aed9ecbef, addrs=[10.31.23.18], sockAddrs=[iZuf62zdiq684kn72aatgjZ/10.31.23.18:47502], discPort=47502, order=37, intOrder=23, lastExchangeTime=1514899147831, loc=true, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false], topVer=37, nodeId8=bfb39f71, msg=null, type=NODE_JOINED, tstamp=1514898728282], crd=TcpDiscoveryNode [id=237a7983-a2d6-4f39-9f39-75cd8b17299c, addrs=[10.26.244.207], sockAddrs=[/10.26.244.207:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1514898728181, loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bfb39f71-fa1d-4547-9b49-b59aed9ecbef, addrs=[10.31.23.18], sockAddrs=[iZuf62zdiq684kn72aatgjZ/10.31.23.18:47502], discPort=47502, order=37, intOrder=23, lastExchangeTime=1514899147831, loc=true, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false], topVer=37, nodeId8=bfb39f71, msg=null, type=NODE_JOINED, tstamp=1514898728282], nodeId=bfb39f71, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=993782641], init=true, lastVer=null, partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], futures=[]], TxReleaseFuture [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], futures=[]], DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], futures=[]]]], exchActions=null, affChangeMsg=null, initTs=1514898728302, centralizedAff=false, changeGlobalStateE=null, done=false, state=SRV, evtLatch=0, remaining=[a364148f-c04f-487e-9935-5298662aa7cc, e7c123fd-5da8-45ca-a04a-65bafe9ed6c6, e417828d-793f-4f24-bf80-7945321d8f51, 237a7983-a2d6-4f39-9f39-75cd8b17299c, bed139ea-9ea9-42e4-b296-79ed51477ad0, d3890e42-6003-4d33-a5f2-52c7c4547f27, 9ece2c56-ced5-4710-be57-94d6bd529a90], 



Previous when we use the DB as backend storage never meet this issue.   attach are those two configuration. 

We have do all the thing, like active the ignite before use it etc. 






aaron@tophold.com
 
From: aaron@tophold.com
Date: 2018-01-02 17:30
To: user
Subject: Re: Re: Some question regards the DataStorageConfiguration when used in multiple data regions.
Hi Denis, 

Thanks for so quick response the NPE as:

Caused by: java.lang.NullPointerException

at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:538) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.metadata(CacheObjectBinaryProcessorImpl.java:194) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1266) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2005) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:284) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:183) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:830) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:845) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.binary.BinaryObjectImpl.field(BinaryObjectImpl.java:308) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:245) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:139) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.H2RowDescriptor.columnValue(H2RowDescriptor.java:303) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue(GridH2KeyValueRowOnheap.java:160) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1132) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1101) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$15800(BPlusTree.java:81) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.find(BPlusTree.java:4597) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5400(BPlusTree.java:4391) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:957) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:938) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find(H2TreeIndex.java:181) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.h2.index.BaseIndex.find(BaseIndex.java:128) ~[h2-1.4.195.jar!/:?]

at org.h2.index.IndexCursor.find(IndexCursor.java:169) ~[h2-1.4.195.jar!/:?]

at org.h2.table.TableFilter.next(TableFilter.java:468) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1452) ~[h2-1.4.195.jar!/:?]

at org.h2.result.LazyResult.hasNext(LazyResult.java:79) ~[h2-1.4.195.jar!/:?]

at org.h2.result.LazyResult.next(LazyResult.java:59) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Select.queryFlat(Select.java:519) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Select.queryWithoutCache(Select.java:625) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Query.query(Query.java:352) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Query.query(Query.java:333) ~[h2-1.4.195.jar!/:?]

at org.h2.command.CommandContainer.query(CommandContainer.java:113) ~[h2-1.4.195.jar!/:?]

at org.h2.command.Command.executeQuery(Command.java:201) ~[h2-1.4.195.jar!/:?]

at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:111) ~[h2-1.4.195.jar!/:?]

at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:961) ~[ignite-indexing-2.3.0.jar!/:2.3.0]




Regards
Aaron

aaron@tophold.com
 
Date: 2018-01-02 16:28
To: user
Subject: Re: Some question regards the DataStorageConfiguration when used in multiple data regions.
Hi Aaron!

Looks like warnings are printed due to a bug. I created a ticket for it: https://issues.apache.org/jira/browse/IGNITE-7347
This bug doesn't affect anything, so you can just ignore the warnings.

You've cut out the most interesting part of the stack trace - place, where NPE occurred. Please provide the full stack trace.

So, this exception appears, when you update a cache from one data region, and query a cache from another? It doesn't sound plausible :) Please provide a reproducer, if it's possible. Or provide more details about your configuration, query entities and a query.

Denis

вт, 2 янв. 2018 г. в 7:12, aaron@tophold.com <aaron@tophold.com>:

hi All, 

Current we are leveraging the DataStorageConfiguration to persist the cache to local file system. 

Can currently the persistence enable flag be set for separated  data Regions? as from our running log. 

Even we only set the region A for persistenceEnabled  but log said:

[WARN ] 2018-01-02 03:27:43.234 [main] [ig] GridCacheDatabaseSharedManager - Page eviction mode for [B_Region] memory region is ignored because Ignite Native Persistence is enabled

[WARN ] 2018-01-02 03:27:43.235 [main] [ig] GridCacheDatabaseSharedManager - Page eviction mode for [C_Region] memory region is ignored because Ignite Native Persistence is enabled


B/C_Region used because some cache refer to those regions. 

Now the problem is if any time we update the data structure in the Other caches in B/C_Region,  this cache in the Region A can not be query any more!


[ERROR] 2018-01-02 03:50:06.328 [query-#202%BookerFX%] [ig] GridMapQueryExecutor - Failed to execute local query.

org.apache.ignite.IgniteCheckedException: Failed to execute SQL query.

at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:968) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:1027) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:1006) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:670) [ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:516) [ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onMessage(GridMapQueryExecutor.java:214) [ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor$2.onMessage(GridMapQueryExecutor.java:174) [ignite-indexing-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:2332) [ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) [ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) [ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) [ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) [ignite-core-2.3.0.jar!/:2.3.0]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=RowSimple [vals=[null, null, null, 'EURUSD', 0]], upper=RowSimple [vals=[null, null, null, 'EURUSD', 4096]]]"; SQL statement:

SELECT

"CacheExample".__Z0._KEY __C0_0,

"CacheExample".__Z0._VAL __C0_1,

__Z0.SEQUENCENUMBER __C0_2

FROM "BookerGenericDomainEventEntryFX".GENERICDOMAINEVENTENTRY __Z0

WHERE (__Z0.SEQUENCENUMBER < ?3) AND ((__Z0.AGGREGATEIDENTIFIER = ?1) AND (__Z0.SEQUENCENUMBER >= ?2))

ORDER BY 3 [50000-195]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.195.jar!/:?]

at org.h2.message.DbException.get(DbException.java:168) ~[h2-1.4.195.jar!/:?]

at org.h2.message.DbException.convert(DbException.java:295) ~[h2-1.4.195.jar!/:?]

at org.h2.command.Command.executeQuery(Command.java:215) ~[h2-1.4.195.jar!/:?]

at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:111) ~[h2-1.4.195.jar!/:?]

at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:961) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

... 14 more

Caused by: org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=RowSimple [vals=[null, null, null, 'EURUSD', 0]], upper=RowSimple [vals=[null, null, null, 'EURUSD', 4096]]]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:965) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:938) ~[ignite-core-2.3.0.jar!/:2.3.0]

at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find(H2TreeIndex.java:181) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

at org.h2.index.BaseIndex.find(BaseIndex.java:128) ~[h2-1.4.195.jar!/:?]

at org.h2.index.IndexCursor.find(IndexCursor.java:169) ~[h2-1.4.195.jar!/:?]

at org.h2.table.TableFilter.next(TableFilter.java:468) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1452) ~[h2-1.4.195.jar!/:?]

at org.h2.result.LazyResult.hasNext(LazyResult.java:79) ~[h2-1.4.195.jar!/:?]

at org.h2.result.LazyResult.next(LazyResult.java:59) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Select.queryFlat(Select.java:519) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Select.queryWithoutCache(Select.java:625) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Query.query(Query.java:352) ~[h2-1.4.195.jar!/:?]

at org.h2.command.dml.Query.query(Query.java:333) ~[h2-1.4.195.jar!/:?]

at org.h2.command.CommandContainer.query(CommandContainer.java:113) ~[h2-1.4.195.jar!/:?]

at org.h2.command.Command.executeQuery(Command.java:201) ~[h2-1.4.195.jar!/:?]

at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:111) ~[h2-1.4.195.jar!/:?]

at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:961) ~[ignite-indexing-2.3.0.jar!/:2.3.0]

... 14 more

Caused by: java.lang.NullPointerException




Cache configuration:

<property name="storagePath" value="${persistent.store.base.path:/var/ignite/persistent}/${app.instance.cnt}/store"/>
<property name="walArchivePath" value="${persistent.store.base.path:/var/ignite/persistent}/${app.instance.cnt}/store/wal/archive"/>
<property name="walPath" value="${persistent.store.base.path:/var/ignite/persistent}/${app.instance.cnt}/store/wal"/>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="A_Region"/>
<property name="initialSize" value="#{2L * 1024L * 1024L * 1024L}"/>
<property name="maxSize" value="#{4L * 1024L * 1024L * 1024L}"/>
<property name="pageEvictionMode" value="DISABLED"/>
<property name="persistenceEnabled" value="true"/>
</bean>


Thanks for your time!



Regards
Aaron