Return-Path: X-Original-To: apmail-ignite-user-archive@minotaur.apache.org Delivered-To: apmail-ignite-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E8DB6196DE for ; Fri, 29 Apr 2016 16:56:15 +0000 (UTC) Received: (qmail 92224 invoked by uid 500); 29 Apr 2016 16:56:15 -0000 Delivered-To: apmail-ignite-user-archive@ignite.apache.org Received: (qmail 92180 invoked by uid 500); 29 Apr 2016 16:56:15 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 92170 invoked by uid 99); 29 Apr 2016 16:56:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2016 16:56:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 52D6E1800E7 for ; Fri, 29 Apr 2016 16:56:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.178 X-Spam-Level: * X-Spam-Status: No, score=1.178 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id wBMKnHreg1hS for ; Fri, 29 Apr 2016 16:56:13 +0000 (UTC) Received: from nm13-vm3.bullet.mail.ne1.yahoo.com (nm13-vm3.bullet.mail.ne1.yahoo.com [98.138.91.143]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 0F8AD5F298 for ; Fri, 29 Apr 2016 16:56:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1461948965; bh=Zcb6PeisldEbxFQs8lZg5oOTmOXYWVwdWgaH8Skppow=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=IRdbHUixLsILjsL3CbwadMf+t3zSB7NuHVOwS2iyyX7SJJLI04dCR8CcMakrrXNJ/07oOzZxZ0Mu9rYuOxIqTpZDMdAk5wFVnB9K0+BUiCIijaizCkCm9A7YgIt3fG7Gd3MQkxZ07xx85M34R9quMLmSP21XIrzPpCq02MXfYdAthiZ9aIUS18PcRuDnDRM368DRDRBKGE2oOtti80IUeZC8KdZT190qIVoytxAscKPHM9JGolzUgSFC2DP9FRcFrs5YvjqDoY644Uhx3Y7k6auADzmxQMXa42c69MkW4tGK3K9ZXaepvQqrhllYjSmEo4xFjr0kakEfWvUMnMwkag== Received: from [98.138.101.128] by nm13.bullet.mail.ne1.yahoo.com with NNFMP; 29 Apr 2016 16:56:05 -0000 Received: from [98.138.226.161] by tm16.bullet.mail.ne1.yahoo.com with NNFMP; 29 Apr 2016 16:56:05 -0000 Received: from [127.0.0.1] by omp1062.mail.ne1.yahoo.com with NNFMP; 29 Apr 2016 16:56:05 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 551431.39359.bm@omp1062.mail.ne1.yahoo.com X-YMail-OSG: .bGhYhkVM1m_22MHKaOw3Hs4AlgAbUtGPvi3yenSWQ0lEtEZAZ7d59C4bXUhJpj LNhHMfspoI5ub3DksDd_1ULvNVtt6umUr26_6HVADwwSygFovyugEmz_ssLUhLZTNrjYolIVz39h wEis_asablREMt1oOEseHWG9Vsq_jiMQhRyIQGoaPJIge7rucL8Dc5zPKvHMnZ36BZF4I4UKjjuG uqDQYP1I9f0.dniB12dKvdUYsE650_UlStC20R.1MUOgkMHOMJhvnG4LtEOXbetQCx3dl7nKY1Ma sCbHmKW.sMe1pT5nFFtwsn2Ezqqh_mWNgnMcSA3U3LKRGBD4iKbqHMum7L4Xo_vOQt7r9EQUmSzq gTq3zPY7HArxaKPpUy5mDmHcCbEJJMUk_nN_QmwNmRbTV3gzFflpBXxHl6fWI0tWwTbyDfLVMBSK 45X8yjI1lIHwPthJ7DX7AfZoHXTVsdlbE8Ah64LVd6MklWoN_PjsHXgVAbv5otOYGzthTZQlqORX 37y_Wh6r3YLAV4yf5EqiF Received: by 98.138.105.250; Fri, 29 Apr 2016 16:56:05 +0000 Date: Fri, 29 Apr 2016 16:56:04 +0000 (UTC) From: vijayendra bhati Reply-To: vijayendra bhati To: "user@ignite.apache.org" Message-ID: <2067518247.3796379.1461948964651.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Re: Number of partitions of IgniteRDD MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3796378_148126923.1461948964643" ------=_Part_3796378_148126923.1461948964643 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable My bad !!! Yea you are right.But now the problem is when I get DataFrame by= using below code and get JavaRDD from it , its number of partitions is 1 a= nd its of type =C2=A0MapPartitionsRDD. String sql =3D "select simulationUUID,stockReturn from STOCKSIMULATIONRETUR= NSVAL where businessDate =3D ? and symbol =3D ?";DataFrame df =3Djic.fromCa= che(PARTITIONED_CACHE_NAME).sql(sql, businessDate,stock);=C2=A0df.javaRDD()= ; This is causing performance issues for me as I have only 1 partition and my= reduceByKey method is not performing in desired way. Regards,Vij =20 On Friday, April 29, 2016 6:11 PM, Vladimir Ozerov wrote: =20 Hi Vij, I see method "getPartitions" in IgniteRDD, not "getNumPartitions". Please c= onfirm that we are talking about the same thing. Anyway, logic of this method is extremely straightforward - it simply call = Ignite.affinity("name_of_your_cache").partitions() method, so it should ret= urn actual number of partitions.=C2=A0"getPartitions" returns array, could = you please show is printed to the console from your code? Vladimir. On Fri, Apr 29, 2016 at 3:10 PM, vijayendra bhati = wrote: Yes its Spark RDD's standard method, but it has been overridden in IgniteRD= D. Regards,Vij=20 On Friday, April 29, 2016 5:25 PM, Vladimir Ozerov wrote: =20 Hi Vij, I am not quite uderstand where does method "getNumPartitions" came from. Is= it on standard Spark API? I do not see it on org.apache.spark.api.java.Jav= aRDD class. Vladimir. On Fri, Apr 29, 2016 at 7:50 AM, vijayendra bhati = wrote: Hi Val, I am creating DataFrame using below code -=C2=A0 public DataFrame getStockSimulationReturnsDataFrame(LocalDate businessDate= ,String stock){ /* * If we use sql query , we are assuming that data is in= cache. * */ String sql =3D "select simulationUUID,stockReturn from STOCKS= IMULATIONRETURNSVAL where businessDate =3D ? and symbol =3D ?"; DataFrame d= f =3Djic.fromCache(PARTITIONED_CACHE_NAME).sql(sql, businessDate,stock); re= turn df; } And to check partitions I am doing -=C2=A0 private JavaRDD getSimulationsForStock(String stock,LocalDate business= Date) { DataFrame df =3D =C2=A0StockSimulationsReaderFactory.getStockSimula= tionStore(jsc, businessDate, businessDate).getStockSimulationReturnsDataFra= me(businessDate, stock); System.out.println("^^^^^^^^"+df.javaRDD().getNumP= artitions()); return df.javaRDD(); } Regards,Vij=20 On Friday, April 29, 2016 3:09 AM, vkulichenko wrote: =20 Hi Vij, How do you check the number of partitions and what are you trying to achieve? Can you show the code? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.co= m/Number-of-partitions-of-IgniteRDD-tp4644p4671.html Sent from the Apache Ignite Users mailing list archive at Nabble.com. =20 =20 ------=_Part_3796378_148126923.1461948964643 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My bad !!!= Yea you are right.But now the problem is when I get DataFrame by using bel= ow code and get JavaRDD from it , its number of partitions is 1 and its of = type  MapPartitionsRDD.

String sql =3D "select simulationUUID,stockReturn from STOCKS= IMULATIONRETURNSVAL where businessDate =3D ? and symbol =3D ?";
DataFrame df =3Djic.fromCache(PARTITIO= NED_CACHE_NAME).sql(sql, businessDate,stock);
 df.java= RDD();

This is causing performance issues for me as= I have only 1 partition and my reduceByKey method is not performing in des= ired way.

=
= Regards,
Vij=



On Friday, April 29, 2016 6:11 PM, Vladimir Ozerov <vozerov@gri= dgain.com> wrote:


Hi Vij,

I see method "getPartitions" in IgniteRDD, not "getNum= Partitions". Please confirm that we are talking about the same thing.
=

Anyway, logic of this method is extremel= y straightforward - it simply call Ignite.affinity("name_of_your_cache").pa= rtitions() method, so it should return actual number of partitions. 
"getPartitions" returns array, could you please show is printed to= the console from your code?


Vladimir.

On Fri, Ap= r 29, 2016 at 3:10 PM, vijayendra bhati <veejayendraa@yahoo.com>= ; wrote:
Yes its Spark RDD's standard method, but it has been o= verridden in IgniteRDD.

Regards,
Vij


<= div style=3D"display:block;">
<= div style=3D"font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, L= ucida Grande, sans-serif;font-size:16px;">
On Friday, April 29, 2016 5:25 PM, Vladimir Ozerov <vozerov@gridgain.com> wrote:


Hi Vij,

<= /div>
I am not quite uderstand where does method "getNumPartitions" cam= e from. Is it on standard Spark API? I do not see it on org.apache.spark= .api.java.JavaRDD class.

Vladim= ir.

On Fri, Apr 29, 2016 at 7:= 50 AM, vijayendra bhati <veejayendraa@yahoo.com> wrote:<= br clear=3D"none">
Hi Val,

I am creating DataFrame using below code - 

=09public DataFrame getStockSimulationReturnsDataFrame(LocalDat= e businessDate,String stock){
=09=09/*
=09=09 * If we use sql query , we are assuming that data is in= cache.
=09=09 * */
=09=09= String sql =3D "select simulationUUID,stockReturn from STOCKSIMULATIONRETUR= NSVAL where businessDate =3D ? and symbol =3D ?";
=09=09DataFrame df =3Djic.fromCache(PA= RTITIONED_CACHE_NAME).sql(sql, businessDate,stock);
=09=09return df;
=09}


<= /div>
And to check partitions I am doing - 

private JavaRDD<Ro= w> getSimulationsForStock(String stock,LocalDate businessDate)
=09{
=09=09DataFrame df = =3D  StockSimulationsReaderFactory.getStockSimulationStore(jsc, busine= ssDate, businessDate).getStockSimulationReturnsDataFrame(businessDate, stoc= k);
=09=09System.out.println("^^^^^^^^"+df.javaRDD().getNumPartitions());
=09=09return df.j= avaRDD();
=09}

Rega= rds,
Vij


On Friday, April 29, 2016 3:09 AM= , vkulichenko <valentin.kulichenko@gmail.com> wrote:


Hi Vij,=


How do you check the number of = partitions and what are you trying to
achieve? Can you sh= ow the code?

-Val


--
View this message in context: http://apache-ignite-users.70518.x= 6.nabble.com/Number-of-partitions-of-IgniteRDD-tp4644p4671.html
Sent from the Apache Ignite Users mailing list archive at Nabble= .com.








------=_Part_3796378_148126923.1461948964643--