Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 08679200C8C for ; Tue, 6 Jun 2017 20:27:33 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 07050160BC6; Tue, 6 Jun 2017 18:27:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 001CE160BB7 for ; Tue, 6 Jun 2017 20:27:31 +0200 (CEST) Received: (qmail 26125 invoked by uid 500); 6 Jun 2017 18:27:31 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 26115 invoked by uid 99); 6 Jun 2017 18:27:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jun 2017 18:27:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id AE7C6CFE62 for ; Tue, 6 Jun 2017 18:27:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.174 X-Spam-Level: ** X-Spam-Status: No, score=2.174 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id XUS3uTeP-dgn for ; Tue, 6 Jun 2017 18:27:29 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id D5FD55FD83 for ; Tue, 6 Jun 2017 18:27:28 +0000 (UTC) Received: from static.162.255.23.37.macminivault.com (unknown [162.255.23.37]) by mwork.nabble.com (Postfix) with ESMTP id 4C0C247D90DDC for ; Tue, 6 Jun 2017 11:27:28 -0700 (MST) Date: Tue, 6 Jun 2017 11:27:28 -0700 (MST) From: baozipu To: user@ignite.apache.org Message-ID: <1496773648306-13432.post@n6.nabble.com> Subject: Error occurs when calling IgniteRDD.savePairs() MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit archived-at: Tue, 06 Jun 2017 18:27:33 -0000 Hello, I am currently writing data Spark RDD to IgniteRDD using IgniteRDD.savePairs() function. I have no problem when writing small amount of data, just like: sharedRDD.savePairs(ss.sparkContext.parallelize(1 to 10000, 64).map(i => (i, i))) However, error occurred when I write a 5Gb data into the IgniteRDD, by calling savePairs() data type of SparkRDD is sparkRDD[Long, Row], type, which Long is just some unique index value, and Row is the org.apache.spark.sql.Row type, containing many elements. Can Ignite able to handle such kind of data type? When I call the savePairs() function, I got all WARNs: WARN GridCachePartitionExchangeManager: Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], node=35ecd3af-484f-4709-978a-58c0d799a636]. Dumping pending objects that might be the cause: 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Ready affinity version: AffinityTopologyVersion [topVer=10, minorTopVer=0] 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=CacheAffinityChangeMessage [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], topVer=10, nodeId8=35ecd3af, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1496772417443]], crd=TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], nodeId=74163cf4, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1496772417443, endTime=1496772417453, ignoreInterrupts=false, state=DONE], init=true, topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=CacheAffinityChangeMessage [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], skipPreload=true, clientOnlyExchange=true, initTs=1496772417443, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[74163cf4-432f-4ed5-8ba3-52c0449d7c52, a8878928-67bd-4b90-82cf-8cd7c10c3abc, e6506f86-18df-4057-8eb4-3a9d5fbcdc41, 490782d6-53b5-4d2b-aca5-8b750a996f87, 0b022f35-5749-41a0-9830-69ae8c17f5a9, 2bf278bb-bafb-4730-8d40-5ab4bdcd164b, dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, 279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, 0ee4105c-907a-4d43-9186-126e08bd783a], srvNodes=[TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=0ee4105c-907a-4d43-9186-126e08bd783a, addrs=[0:0:0:0:0:0:0:1%1, 10.93.128.99, 10.93.136.99, 127.0.0.1], sockAddrs=[mapr417.sv.aaa.com/10.93.128.99:47500, /127.0.0.1:47500, mapr417-eth1.sv.aaa.com/10.93.136.99:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1496771432376, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=490782d6-53b5-4d2b-aca5-8b750a996f87, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.34.21], sockAddrs=[sv1-hp0201-04.sv.aaa.com/172.30.34.21:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=4, intOrder=4, lastExchangeTime=1496772399342, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.169], sockAddrs=[sv1-hp0213-20.sv.aaa.com/172.30.33.169:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=5, intOrder=5, lastExchangeTime=1496772404498, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=a8878928-67bd-4b90-82cf-8cd7c10c3abc, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.59], sockAddrs=[sv1-hp0203-10.sv.aaa.com/172.30.32.59:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1496772404641, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=2bf278bb-bafb-4730-8d40-5ab4bdcd164b, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.140], sockAddrs=[/127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, sv1-hp0212-11.sv.aaa.com/172.30.33.140:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1496772404824, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=e6506f86-18df-4057-8eb4-3a9d5fbcdc41, addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.11, 10.93.136.244, 127.0.0.1], sockAddrs=[mapr354-eth1.sv.aaa.com/10.93.136.244:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, mapr354.sv.aaa.com/10.93.131.11:47500], discPort=47500, order=8, intOrder=8, lastExchangeTime=1496772406203, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=0b022f35-5749-41a0-9830-69ae8c17f5a9, addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.135, 10.93.137.135, 127.0.0.1], sockAddrs=[mapr260-eth1.sv.aaa.com/10.93.137.135:47501, /127.0.0.1:47501, /0:0:0:0:0:0:0:1%1:47501, mapr260.sv.aaa.com/10.93.131.135:47501], discPort=47501, order=9, intOrder=9, lastExchangeTime=1496772406254, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.188, 10.93.137.165, 127.0.0.1], sockAddrs=[bfd-proda059-eth1.sv.aaa.com/10.93.137.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, mapr379.sv.aaa.com/10.93.131.188:47500], discPort=47500, order=10, intOrder=10, lastExchangeTime=1496772407085, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1496772417443, endTime=0, ignoreInterrupts=false, state=INIT]] 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Pending exchange futures: 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Last 10 exchange futures (total: 11): 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=CacheAffinityChangeMessage [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], topVer=10, nodeId8=35ecd3af, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1496772417443]], crd=TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], nodeId=74163cf4, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1496772417443, endTime=1496772417453, ignoreInterrupts=false, state=DONE], init=true, topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=CacheAffinityChangeMessage [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], skipPreload=true, clientOnlyExchange=true, initTs=1496772417443, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[74163cf4-432f-4ed5-8ba3-52c0449d7c52, a8878928-67bd-4b90-82cf-8cd7c10c3abc, e6506f86-18df-4057-8eb4-3a9d5fbcdc41, 490782d6-53b5-4d2b-aca5-8b750a996f87, 0b022f35-5749-41a0-9830-69ae8c17f5a9, 2bf278bb-bafb-4730-8d40-5ab4bdcd164b, dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, 279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, 0ee4105c-907a-4d43-9186-126e08bd783a], srvNodes=[TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=0ee4105c-907a-4d43-9186-126e08bd783a, addrs=[0:0:0:0:0:0:0:1%1, 10.93.128.99, 10.93.136.99, 127.0.0.1], sockAddrs=[mapr417.sv.aaa.com/10.93.128.99:47500, /127.0.0.1:47500, mapr417-eth1.sv.aaa.com/10.93.136.99:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1496771432376, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=490782d6-53b5-4d2b-aca5-8b750a996f87, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.34.21], sockAddrs=[sv1-hp0201-04.sv.aaa.com/172.30.34.21:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=4, intOrder=4, lastExchangeTime=1496772399342, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.169], sockAddrs=[sv1-hp0213-20.sv.aaa.com/172.30.33.169:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=5, intOrder=5, lastExchangeTime=1496772404498, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=a8878928-67bd-4b90-82cf-8cd7c10c3abc, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.59], sockAddrs=[sv1-hp0203-10.sv.aaa.com/172.30.32.59:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=6, intOrder=6, lastExchangeTime=1496772404641, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=2bf278bb-bafb-4730-8d40-5ab4bdcd164b, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.140], sockAddrs=[/127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, sv1-hp0212-11.sv.aaa.com/172.30.33.140:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1496772404824, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=e6506f86-18df-4057-8eb4-3a9d5fbcdc41, addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.11, 10.93.136.244, 127.0.0.1], sockAddrs=[mapr354-eth1.sv.aaa.com/10.93.136.244:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, mapr354.sv.aaa.com/10.93.131.11:47500], discPort=47500, order=8, intOrder=8, lastExchangeTime=1496772406203, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=0b022f35-5749-41a0-9830-69ae8c17f5a9, addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.135, 10.93.137.135, 127.0.0.1], sockAddrs=[mapr260-eth1.sv.aaa.com/10.93.137.135:47501, /127.0.0.1:47501, /0:0:0:0:0:0:0:1%1:47501, mapr260.sv.aaa.com/10.93.131.135:47501], discPort=47501, order=9, intOrder=9, lastExchangeTime=1496772406254, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], TcpDiscoveryNode [id=279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.188, 10.93.137.165, 127.0.0.1], sockAddrs=[bfd-proda059-eth1.sv.aaa.com/10.93.137.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, mapr379.sv.aaa.com/10.93.131.188:47500], discPort=47500, order=10, intOrder=10, lastExchangeTime=1496772407085, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1496772417443, endTime=0, ignoreInterrupts=false, state=INIT]] What is about? I use Ignite 1.9 -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Error-occurs-when-calling-IgniteRDD-savePairs-tp13432.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.