Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BE7752007D1 for ; Thu, 12 May 2016 10:13:16 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BD4AC160939; Thu, 12 May 2016 08:13:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B8DB5160868 for ; Thu, 12 May 2016 10:13:15 +0200 (CEST) Received: (qmail 53356 invoked by uid 500); 12 May 2016 08:13:14 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 53346 invoked by uid 99); 12 May 2016 08:13:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2016 08:13:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 28556180538 for ; Thu, 12 May 2016 08:13:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.998 X-Spam-Level: * X-Spam-Status: No, score=1.998 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id ix01v2aD2Lg7 for ; Thu, 12 May 2016 08:13:12 +0000 (UTC) Received: from GWS08.hcl.com (gws08.hcl.com [192.8.186.131]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B2D385F247 for ; Thu, 12 May 2016 08:13:10 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.24,609,1454956200"; d="scan'208,217";a="50780261" Received: from unknown (HELO CHN-CORP-HT02.CORP.HCL.IN) ([10.249.2.34]) by GWS08.hcl.com with ESMTP/TLS/AES128-SHA; 12 May 2016 13:43:01 +0530 Received: from CHN-HCLT-CSHT07.HCLT.CORP.HCL.IN (10.150.12.32) by CHN-CORP-HT02.CORP.HCL.IN (10.249.2.34) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 12 May 2016 13:42:33 +0530 Received: from CHN-HCLT-MBS07.HCLT.CORP.HCL.IN ([10.150.12.22]) by CHN-HCLT-CSHT07.HCLT.CORP.HCL.IN ([::1]) with mapi id 14.03.0235.001; Thu, 12 May 2016 13:42:33 +0530 From: Vignesh Irulappan To: "user@ignite.apache.org" Subject: How to cache Spark Dataframe in Apache ignite Thread-Topic: How to cache Spark Dataframe in Apache ignite Thread-Index: AdGsJgf3pAzyjHhvSIy0gDNBURlFng== Date: Thu, 12 May 2016 08:12:32 +0000 Message-ID: <312700F59A98294C8C00D4A258060AA92AA1E2DA@chn-hclt-mbs07.HCLT.CORP.HCL.IN> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.150.5.114] Content-Type: multipart/alternative; boundary="_000_312700F59A98294C8C00D4A258060AA92AA1E2DAchnhcltmbs07HCL_" MIME-Version: 1.0 archived-at: Thu, 12 May 2016 08:13:16 -0000 --_000_312700F59A98294C8C00D4A258060AA92AA1E2DAchnhcltmbs07HCL_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hello, I am writing a code to cache RDBMS data using spark SQLContext JDBC connect= ion. Once a Dataframe is created I want to cache that reusltset using apach= e ignite thereby making other applications to make use of the resultset. He= re is the code snippet. object test { def main(args:Array[String]) { val configuration =3D new Configuration() val config=3D"src/main/scala/config.xml" val sparkConf =3D new SparkConf().setAppName("test").setMaster("local= [*]") val sc=3Dnew SparkContext(sparkConf) val sqlContext =3D new org.apache.spark.sql.SQLContext(sc) val sql_dump1=3DsqlContext.read.format("jdbc").option("url", "jdbc UR= L").option("driver", "com.mysql.jdbc.Driver").option("dbtable", mysql_table= _statement).option("user", "username").option("password", "pass").load() val ic =3D new IgniteContext[Integer, Integer](sc, config) val sharedrdd =3D ic.fromCache("hbase_metadata") //How to cache sql_dump1 dataframe } } Now the question is how to cache a dataframe, IgniteRDD has savepairs metho= d but it accepts key and value as RDD[Integer], but I have a dataframe even= if I convert that to RDD i would only be getting RDD[Row]. The savepairs m= ethod consisting of RDD of Integer more specific what if I have a string of= RDD as value? Is it good to cache dataframe or any other better approach t= o cache the resultset. Thanks and Regards, Vignesh ::DISCLAIMER:: ---------------------------------------------------------------------------= ------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and inte= nded for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as informa= tion could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in trans= mission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability = on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the = author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, disse= mination, copying, disclosure, modification, distribution and / or publication of this message without the prior written= consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please= delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses = and other defects. ---------------------------------------------------------------------------= ------------------------------------------------------------------------- --_000_312700F59A98294C8C00D4A258060AA92AA1E2DAchnhcltmbs07HCL_ Content-Type: text/html; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable

Hello,

 

I am writing a code to cache RDBMS data= using spark SQLContext JDBC connection. Once a Dataframe is created I want= to cache that reusltset using apache ignite thereby making other applications to make use of the resultset. Here is the code s= nippet.

 

object test

{

 

  def main(args:Array[String])

  {

 

      val configuration =3D = new Configuration()

      val config=3D"src= /main/scala/config.xml"

    

      val sparkConf =3D= new SparkConf().setAppName("test").setMaster("local[*]"= ;)

      val sc=3Dnew SparkCont= ext(sparkConf)

      val sqlContext =3D new= org.apache.spark.sql.SQLContext(sc)

      val sql_dump1=3DsqlCon= text.read.format("jdbc").option("url", "jdbc URL&q= uot;).option("driver", "com.mysql.jdbc.Driver").option(= "dbtable", mysql_table_statement).option("user", "username").option("password", "pass").load(= )

     

      val ic =3D new Ig= niteContext[Integer, Integer](sc, config)

      

      val sharedrdd =3D= ic.fromCache("hbase_metadata")

      

      //How to cache sq= l_dump1 dataframe

 

  }

}

 

Now the question is how to cache a data= frame, IgniteRDD has savepairs method but it accepts key and value as RDD[I= nteger], but I have a dataframe even if I convert that to RDD i would only be getting RDD[Row]. The savepairs method consist= ing of RDD of Integer more specific what if I have a string of RDD as value= ? Is it good to cache dataframe or any other better approach to cache the r= esultset.

 

Thanks and Regards,

Vignesh



::DISCLAIMER= ::
---------------------------------------------------------------------= ---------------------------------------------------------------------------= ----

The contents of thi= s e-mail and any attachment(s) are confidential and intended for the named = recipient(s) only.
E-mail transmission is not guaranteed to be secure or= error-free as information could be intercepted, corrupted,
lost, destr= oyed, arrive late or incomplete, or may contain viruses in transmission. Th= e e mail and its contents
(with or without referred errors) shall there= fore not attach any liability on the originator or HCL or its affiliates. <= br>Views or opinions, if any, presented in this email are solely those of t= he author and may not necessarily reflect the
views or opinions of HCL = or its affiliates. Any form of reproduction, dissemination, copying, disclo= sure, modification,
distribution and / or publication of this message w= ithout the prior written consent of authorized representative of
HCL is= strictly prohibited. If you have received this email in error please delet= e it and notify the sender immediately.
Before opening any email and/or= attachments, please check them for viruses and other defects.

-------------------= ---------------------------------------------------------------------------= ------------------------------------------------------

--_000_312700F59A98294C8C00D4A258060AA92AA1E2DAchnhcltmbs07HCL_--