ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From F7753 <mabiao...@foxmail.com>
Subject Re: A question of ignite on spark streaming
Date Fri, 01 Apr 2016 01:27:57 GMT
Thanks for your advice ,that helps me a lot.I've received a mail list below,
is there something wrong?
Hi. This is the qmail-send program at apache.org.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

Must be sent from an @apache.org address or a subscriber address or an
address in LDAP.

--- Below this line is a copy of the message.

Return-Path: <mabiaocsu@foxmail.com>
Received: (qmail 17052 invoked by uid 99); 31 Mar 2016 10:18:36 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO
spamd4-us-west.apache.org) (
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2016 10:18:36 +0000
Received: from localhost (localhost [])
by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org)
with ESMTP id DC2ADC0222
for <user@ignite.apache.org>; Thu, 31 Mar 2016 10:18:35 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 1.772
X-Spam-Level: *
X-Spam-Status: No, score=1.772 tagged_above=-999 required=6.31
SPF_SOFTFAIL=0.972] autolearn=disabled
Received: from mx2-lw-eu.apache.org ([])
by localhost (spamd4-us-west.apache.org []) (amavisd-new, port
with ESMTP id 1WFzTO9CbMCJ for <user@ignite.apache.org>;
Thu, 31 Mar 2016 10:18:34 +0000 (UTC)
Received: from mbob.nabble.com (mbob.nabble.com [])
by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTP
for <user@ignite.apache.org>; Thu, 31 Mar 2016 10:18:34 +0000 (UTC)
Received: from malf.nabble.com (unknown [])
by mbob.nabble.com (Postfix) with ESMTP id 1CA3F24242A1
for <user@ignite.apache.org>; Thu, 31 Mar 2016 03:07:13 -0700 (PDT)
Date: Thu, 31 Mar 2016 03:05:10 -0700 (PDT)
From: F7753 <mabiaocsu@foxmail.com>
To: user@ignite.apache.org
Message-ID: <1459418710613-3815.post@n6.nabble.com>
Subject: How to use ignite on spark streaming to join two tables in ignite
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

I want to use two igniteRDD to cache dataFrames: one is a static table, and
the other is a stream table which the data comes from kafka or socket and
But the IgniteContext has only two parameters so that I can only create one
cache. Below are the details:

This is the static table ,the data has already known
val smallTableDF = sqlCtx.createDataFrame(smallTableData.rdd,

// create ignite context (embeded mode)
val igniteContext = new IgniteContext[String, staticTb](ssc.sparkContext, ()
=> new
    IgniteConfiguration().setPeerClassLoadingEnabled(true), false)

// small table cache config
val smallTableCacheCfg = new CacheConfiguration[String, staticTb]()
smallTableCacheCfg.setIndexedTypes(classOf[String], classOf[staticTb]) //
table has "staticTb" name
This is the stream table , the data comes from socket
val sourceTableCacheCfg = new CacheConfiguration[String, streamTb]()
sourceTableCacheCfg.setIndexedTypes(classOf[String], classOf[streamTb]) //
table has "streamTb" name

I want to write a join sql to get some result, But I can not create the
second cache
// ignite source table cache
val sourceTableCache = igniteContext.fromCache(sourceTableCacheCfg)

Warmest Regards

View this message in context:
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/A-question-of-ignite-on-spark-streaming-tp3815p3835.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message