ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From baozipu <job.lei.zh...@gmail.com>
Subject Error occurs when calling IgniteRDD.savePairs()
Date Tue, 06 Jun 2017 18:27:28 GMT
Hello, I am currently writing data Spark RDD to IgniteRDD using
IgniteRDD.savePairs() function.
I have no problem when writing small amount of data, just like:

sharedRDD.savePairs(ss.sparkContext.parallelize(1 to 10000, 64).map(i => (i,
i)))

However, error occurred when I write a 5Gb data into the IgniteRDD, by
calling savePairs()

data type of SparkRDD is  sparkRDD[Long, Row], type, which Long is just some
unique index value, and Row is the org.apache.spark.sql.Row type, containing
many elements.

Can Ignite able to handle such kind of data type?

When I call the savePairs() function, I got all WARNs:

WARN GridCachePartitionExchangeManager: Failed to wait for partition map
exchange [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1],
node=35ecd3af-484f-4709-978a-58c0d799a636]. Dumping pending objects that
might be the cause: 

17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Ready affinity
version: AffinityTopologyVersion [topVer=10, minorTopVer=0]

17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Last exchange
future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryCustomEvent
[customMsg=CacheAffinityChangeMessage
[id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61,
topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion
[topVer=10, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.33.165],
sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], topVer=10, nodeId8=35ecd3af, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1496772417443]], crd=TcpDiscoveryNode
[id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.33.165],
sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], nodeId=74163cf4,
evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2,
res=true, startTime=1496772417443, endTime=1496772417453,
ignoreInterrupts=false, state=DONE], init=true, topSnapshot=null,
lastVer=null, partReleaseFut=null, affChangeMsg=CacheAffinityChangeMessage
[id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61,
topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], skipPreload=true,
clientOnlyExchange=true, initTs=1496772417443, centralizedAff=false,
forcedRebFut=null, evtLatch=0,
remaining=[74163cf4-432f-4ed5-8ba3-52c0449d7c52,
a8878928-67bd-4b90-82cf-8cd7c10c3abc, e6506f86-18df-4057-8eb4-3a9d5fbcdc41,
490782d6-53b5-4d2b-aca5-8b750a996f87, 0b022f35-5749-41a0-9830-69ae8c17f5a9,
2bf278bb-bafb-4730-8d40-5ab4bdcd164b, dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7,
279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, 0ee4105c-907a-4d43-9186-126e08bd783a],
srvNodes=[TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165],
sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=0ee4105c-907a-4d43-9186-126e08bd783a,
addrs=[0:0:0:0:0:0:0:1%1, 10.93.128.99, 10.93.136.99, 127.0.0.1],
sockAddrs=[mapr417.sv.aaa.com/10.93.128.99:47500, /127.0.0.1:47500,
mapr417-eth1.sv.aaa.com/10.93.136.99:47500, /0:0:0:0:0:0:0:1%1:47500],
discPort=47500, order=2, intOrder=2, lastExchangeTime=1496771432376,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
TcpDiscoveryNode [id=490782d6-53b5-4d2b-aca5-8b750a996f87,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.34.21],
sockAddrs=[sv1-hp0201-04.sv.aaa.com/172.30.34.21:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=4, intOrder=4,
lastExchangeTime=1496772399342, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.169],
sockAddrs=[sv1-hp0213-20.sv.aaa.com/172.30.33.169:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=5, intOrder=5,
lastExchangeTime=1496772404498, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=a8878928-67bd-4b90-82cf-8cd7c10c3abc,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.59],
sockAddrs=[sv1-hp0203-10.sv.aaa.com/172.30.32.59:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=6, intOrder=6,
lastExchangeTime=1496772404641, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=2bf278bb-bafb-4730-8d40-5ab4bdcd164b,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.140],
sockAddrs=[/127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500,
sv1-hp0212-11.sv.aaa.com/172.30.33.140:47500], discPort=47500, order=7,
intOrder=7, lastExchangeTime=1496772404824, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode
[id=e6506f86-18df-4057-8eb4-3a9d5fbcdc41, addrs=[0:0:0:0:0:0:0:1%1,
10.93.131.11, 10.93.136.244, 127.0.0.1],
sockAddrs=[mapr354-eth1.sv.aaa.com/10.93.136.244:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500, mapr354.sv.aaa.com/10.93.131.11:47500],
discPort=47500, order=8, intOrder=8, lastExchangeTime=1496772406203,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
TcpDiscoveryNode [id=0b022f35-5749-41a0-9830-69ae8c17f5a9,
addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.135, 10.93.137.135, 127.0.0.1],
sockAddrs=[mapr260-eth1.sv.aaa.com/10.93.137.135:47501, /127.0.0.1:47501,
/0:0:0:0:0:0:0:1%1:47501, mapr260.sv.aaa.com/10.93.131.135:47501],
discPort=47501, order=9, intOrder=9, lastExchangeTime=1496772406254,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
TcpDiscoveryNode [id=279e2d61-05f7-4cdb-ac7f-6f4e1c8518de,
addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.188, 10.93.137.165, 127.0.0.1],
sockAddrs=[bfd-proda059-eth1.sv.aaa.com/10.93.137.165:47500,
/127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500,
mapr379.sv.aaa.com/10.93.131.188:47500], discPort=47500, order=10,
intOrder=10, lastExchangeTime=1496772407085, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], super=GridFutureAdapter
[resFlag=0, res=null, startTime=1496772417443, endTime=0,
ignoreInterrupts=false, state=INIT]]


17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Pending exchange
futures:
17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Last 10 exchange
futures (total: 11):
17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: >>>
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryCustomEvent
[customMsg=CacheAffinityChangeMessage
[id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61,
topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion
[topVer=10, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.33.165],
sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], topVer=10, nodeId8=35ecd3af, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1496772417443]], crd=TcpDiscoveryNode
[id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.33.165],
sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], nodeId=74163cf4,
evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2,
res=true, startTime=1496772417443, endTime=1496772417453,
ignoreInterrupts=false, state=DONE], init=true, topSnapshot=null,
lastVer=null, partReleaseFut=null, affChangeMsg=CacheAffinityChangeMessage
[id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61,
topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], skipPreload=true,
clientOnlyExchange=true, initTs=1496772417443, centralizedAff=false,
forcedRebFut=null, evtLatch=0,
remaining=[74163cf4-432f-4ed5-8ba3-52c0449d7c52,
a8878928-67bd-4b90-82cf-8cd7c10c3abc, e6506f86-18df-4057-8eb4-3a9d5fbcdc41,
490782d6-53b5-4d2b-aca5-8b750a996f87, 0b022f35-5749-41a0-9830-69ae8c17f5a9,
2bf278bb-bafb-4730-8d40-5ab4bdcd164b, dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7,
279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, 0ee4105c-907a-4d43-9186-126e08bd783a],
srvNodes=[TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165],
sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=0ee4105c-907a-4d43-9186-126e08bd783a,
addrs=[0:0:0:0:0:0:0:1%1, 10.93.128.99, 10.93.136.99, 127.0.0.1],
sockAddrs=[mapr417.sv.aaa.com/10.93.128.99:47500, /127.0.0.1:47500,
mapr417-eth1.sv.aaa.com/10.93.136.99:47500, /0:0:0:0:0:0:0:1%1:47500],
discPort=47500, order=2, intOrder=2, lastExchangeTime=1496771432376,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
TcpDiscoveryNode [id=490782d6-53b5-4d2b-aca5-8b750a996f87,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.34.21],
sockAddrs=[sv1-hp0201-04.sv.aaa.com/172.30.34.21:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=4, intOrder=4,
lastExchangeTime=1496772399342, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.169],
sockAddrs=[sv1-hp0213-20.sv.aaa.com/172.30.33.169:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=5, intOrder=5,
lastExchangeTime=1496772404498, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=a8878928-67bd-4b90-82cf-8cd7c10c3abc,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.59],
sockAddrs=[sv1-hp0203-10.sv.aaa.com/172.30.32.59:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=6, intOrder=6,
lastExchangeTime=1496772404641, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], TcpDiscoveryNode [id=2bf278bb-bafb-4730-8d40-5ab4bdcd164b,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.140],
sockAddrs=[/127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500,
sv1-hp0212-11.sv.aaa.com/172.30.33.140:47500], discPort=47500, order=7,
intOrder=7, lastExchangeTime=1496772404824, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode
[id=e6506f86-18df-4057-8eb4-3a9d5fbcdc41, addrs=[0:0:0:0:0:0:0:1%1,
10.93.131.11, 10.93.136.244, 127.0.0.1],
sockAddrs=[mapr354-eth1.sv.aaa.com/10.93.136.244:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500, mapr354.sv.aaa.com/10.93.131.11:47500],
discPort=47500, order=8, intOrder=8, lastExchangeTime=1496772406203,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
TcpDiscoveryNode [id=0b022f35-5749-41a0-9830-69ae8c17f5a9,
addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.135, 10.93.137.135, 127.0.0.1],
sockAddrs=[mapr260-eth1.sv.aaa.com/10.93.137.135:47501, /127.0.0.1:47501,
/0:0:0:0:0:0:0:1%1:47501, mapr260.sv.aaa.com/10.93.131.135:47501],
discPort=47501, order=9, intOrder=9, lastExchangeTime=1496772406254,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
TcpDiscoveryNode [id=279e2d61-05f7-4cdb-ac7f-6f4e1c8518de,
addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.188, 10.93.137.165, 127.0.0.1],
sockAddrs=[bfd-proda059-eth1.sv.aaa.com/10.93.137.165:47500,
/127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500,
mapr379.sv.aaa.com/10.93.131.188:47500], discPort=47500, order=10,
intOrder=10, lastExchangeTime=1496772407085, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], super=GridFutureAdapter
[resFlag=0, res=null, startTime=1496772417443, endTime=0,
ignoreInterrupts=false, state=INIT]]

What is about? I use Ignite 1.9



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Error-occurs-when-calling-IgniteRDD-savePairs-tp13432.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message