ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From userx <gagan...@gmail.com>
Subject javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Some of DataStreamer operations failed [failedCount=1]
Date Fri, 28 Dec 2018 15:09:39 GMT
Hi

I have recently upgraded to 2.7 and are in the process of load testing the
ignite infrastructure. In order to do so we have fired request which
populates the data to these ignite caches. We have got a lot of errors with
the message mentioned in the request.

Here is the excerpt from ignite started in client mode from a java program.
Caused by: javax.cache.CacheException: class
org.apache.ignite.IgniteCheckedException: Some of DataStreamer operations
failed [failedCount=1]
	at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
	at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1287)
	at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1388)
	at com.XXX.datagrid.DataGridClient.writeAll(DataGridClient.java:190)



I am also pasting the ignite server logs
2018-12-28 14:52:55,885 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.exchange.time - Started exchange init
[topVer=AffinityTopologyVersion [topVer=60, minorTopVer=1],
mvccCrd=MvccCoordinator [nodeId=a54f6008-64d4-4aea-aa26-0a716c29c1db,
crdVer=1546005033221, topVer=AffinityTopologyVersion [topVer=15,
minorTopVer=0]], mvccCrdChange=false, crd=true, evt=DISCOVERY_CUSTOM_EVT,
evtNode=cdea7a3b-7b3f-431c-aa90-3fe44de336ea,
customEvt=DynamicCacheChangeBatch
[id=725a415f761-0c9822c2-ec75-433a-b460-6b941dec650d,
reqs=[DynamicCacheChangeRequest [cacheName=6532_40789, hasCfg=true,
nodeId=cdea7a3b-7b3f-431c-aa90-3fe44de336ea, clientStartOnly=false,
stop=false, destroy=false, disabledAfterStartfalse]],
exchangeActions=ExchangeActions [startCaches=[6532_40789], stopCaches=null,
startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null],
startCaches=false], allowMerge=false]
2018-12-28 14:52:55,886 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager -
Updating caches registry performed in 1 ms.
2018-12-28 14:52:55,895 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.GridCacheProcessor - Started
cache [name=6532_40789, id=-443798693, group=ProductCache,
memoryPolicyName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=1,
mvcc=false], encryptionEnabled=false]
2018-12-28 14:52:55,896 WARN
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager - No
server nodes found for cache client: 6532_40789
2018-12-28 14:52:55,896 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager -
Caches starting performed in 10 ms.
2018-12-28 14:52:55,896 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager -
Affinity initialization for started caches performed in 0 ms.
2018-12-28 14:52:55,896 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=60, minorTopVer=1], waitTime=0ms,
futInfo=NA, mode=DISTRIBUTED]
2018-12-28 14:52:55,897 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Finished waiting for partitions release latch: ServerLatch [permits=0,
pendingAcks=[], super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=60, minorTopVer=1]]]
2018-12-28 14:52:55,897 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=60, minorTopVer=1], waitTime=0ms,
futInfo=NA, mode=LOCAL]
2018-12-28 14:52:55,897 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager
- Logical recovery performed in 0 ms.
2018-12-28 14:52:55,897 INFO
[exchange-worker-#123%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.exchange.time - Finished exchange init
[topVer=AffinityTopologyVersion [topVer=60, minorTopVer=1], crd=true]
2018-12-28 14:52:55,903 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Coordinator received single message [ver=AffinityTopologyVersion
[topVer=60, minorTopVer=1], node=401772af-63d0-4bd4-b6b7-7199f53ed270,
allReceived=true]
2018-12-28 14:52:55,903 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=60,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=60, minorTopVer=1]]
2018-12-28 14:52:55,903 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Affinity changes (coordinator) applied in 0 ms.
2018-12-28 14:52:55,904 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Partitions validation performed in 1 ms.
2018-12-28 14:52:55,904 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Partitions assignment performed in 0 ms.
2018-12-28 14:52:55,906 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Preparing Full Message performed in 2 ms.
2018-12-28 14:52:55,907 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Sending Full Message performed in 1 ms.
2018-12-28 14:52:55,907 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Sending Full Message to all nodes performed in 1 ms.
2018-12-28 14:52:55,907 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Finish exchange future [startVer=AffinityTopologyVersion [topVer=60,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=60, minorTopVer=1],
err=null]
2018-12-28 14:52:55,907 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager
- Finished indexes rebuilding for cache [name=6532_40789,
grpName=ProductCache]
2018-12-28 14:52:55,907 INFO
[sys-#752%f3a9610b-2b22-4323-92fe-2c9bf79200f0%] {}
org.apache.ignite.internal.processors.cache.GridCacheProcessor - Finish
proxy initialization, cacheName=6532_40789,
localNodeId=a54f6008-64d4-4aea-aa26-0a716c29c1db


Please ignore the topVer=60 as we stop the ignite client to let the GC clean
the objects of the streamer.  The max topology is 23 with 20 clients and 3
servers. But since we start getting these failures, we close the ignite
client which stops the node and then it leaves a topology which increases
the topology version.

Streamer code - Pasting relevant code
    try (IgniteDataStreamer<BenchmarkCacheStoreKey, OptionalDouble> streamer
= ignite.dataStreamer(cacheName)) {
      LOGGER.info("IgniteDataStreamer initialized.");
      LOGGER.info("IgniteDataStreamer setting time out (close or flush
opertations) to "+streamerTimeOut+" ms");
      streamer.timeout(streamerTimeOut);
      LOGGER.info("Starting to add data for a map of size "+map.size());
      streamer.addData(map);
      LOGGER.info("Starting to flush data and wait for maximum
"+streamerTimeOut+" ms before timing out for a map of size "+map.size());
    } catch(Exception e) {
      LOGGER.info("An exception came while writing to cache with name :-"+
cacheName);
      e.printStackTrace();
      if (igniteClient != null) {
        LOGGER.info("Stopping the ignite client to clean redundant
objetcts.");
        igniteClient.stopIgniteClient();
      }
      throw new IgniteException(e);
    }finally {
      LOGGER.info("Finished streaming data.");
    }

Can someone help me on the same ?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message