ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shawn.du" <shawn...@neulion.com.cn>
Subject Re: compute ignite data with spark
Date Tue, 27 Feb 2018 03:50:23 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.7;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.7;'>
    <div>
<div>Hi, stan,</div><div><br></div><div>Thanks. I am evaluating
igniteRDD. it is cool! &nbsp;I am new to spark, I can get sqlContext by ds.sqlContext.</div><div>If
I want to query all using select, what's the table name?</div><div>see below in
spark-shell, &nbsp;I want to do something like&nbsp;</div><div>&gt;
sc.sql("select * from "). &nbsp;Where is the table name?</div><div>================================================</div><div><div>scala&gt;val
rdd = ic.fromCache[String,BinaryObject]("testCache")</div><div>rdd: org.apache.ignite.spark.IgniteRDD[String,org.apache.ignite.binary.BinaryObject]
= IgniteRDD[0] at RDD at IgniteAbstractRDD.scala:32</div><div>scala&gt; val
ds = rdd.sql("select site,timestamp,product from testCache where site=?",("site1"))</div><div>ds:
org.apache.spark.sql.DataFrame = [SITE: string, TIMESTAMP: bigint ... 1 more field]</div><div>scala&gt;
ds.schema</div><div>res1: org.apache.spark.sql.types.StructType = StructType(StructField(SITE,StringType,true),
StructField(TIMESTAMP,LongType,true), StructField(PRODUCT,StringType,true))</div><div>scala&gt;
val sc = ds.sqlContext</div><div>sc: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@78682201</div></div><div><br></div>
<div id="ntes-pcmail-signature" style="font-family:'微软雅黑'">
    <style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #199cff;
            cursor: pointer;
        }
        a#ntes-pcmail-signature-default:active {
            text-decoration: underline;
            color: #246fce;
            cursor: pointer;
        }
    </style>

                <div style="font-size:14px; padding: 0;  margin:0;">
                    <div style="font-family:&quot;微软雅黑&quot;; font-size:
13px; color:#000000">
    <style>
        font{
            line-height: 1.5;
        }
    </style>
<div id="ntes-pcmail-signature-default" style="font-size:14px; color:#000; text-decoration:
none;">Thanks</div><div id="ntes-pcmail-signature-default" style="font-size:14px;
color:#000; text-decoration: none;">Shawn</div>
</div>
                </div>

</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">2/26/2018 18:25</span>,<a class="mail-to" style="text-decoration:none;color:#2a83f2;"
href="mailto:stanlukyanov@gmail.com">Stanislav Lukyanov&lt;stanlukyanov@gmail.com&gt;</a>
wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family:
'微软雅黑';">
<style type="text/css"><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Microsoft YaHei";
	panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
	{font-family:"\@Microsoft YaHei";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:2.0cm 42.5pt 2.0cm 3.0cm;}
div.WordSection1
	{page:WordSection1;}
-->
</style><div class="WordSection1"><p class="MsoNormal"><span lang="EN-US">Hi
Shawn,<o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><o:p>&nbsp;</o:p></span></p><p
class="MsoNormal"><span lang="EN-US">You can use Ignite standalone and you can also
use it together with Spark.<o:p></o:p></span></p><p class="MsoNormal"><span
lang="EN-US">Please take a look at these SO question and an article:<o:p></o:p></span></p><p
class="MsoNormal"><a href="https://stackoverflow.com/questions/36036910/apache-spark-vs-apache-ignite">https://stackoverflow.com/questions/36036910/apache-spark-vs-apache-ignite</a></p><p
class="MsoNormal"><a href="https://insidebigdata.com/2016/06/20/apache-ignite-and-apache-spark-complementary-in-memory-computing-solutions/">https://insidebigdata.com/2016/06/20/apache-ignite-and-apache-spark-complementary-in-memory-computing-solutions/</a></p><p
class="MsoNormal"><o:p>&nbsp;</o:p></p><p class="MsoNormal"><span
lang="EN-US">Stan</span></p><p class="MsoNormal"><o:p>&nbsp;</o:p></p><div
style="mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt
0cm 0cm 0cm"><p class="MsoNormal" style="border:none;padding:0cm"><b>From:
</b><a href="mailto:shawn.du@neulion.com.cn">shawn.du</a><br><b>Sent:
</b>24 февраля 2018 г. 9:56<br><b>To: </b><a href="mailto:user@ignite.apache.org">user</a><br><b>Subject:
</b>compute ignite data with spark</p></div><p class="MsoNormal"><o:p>&nbsp;</o:p></p><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black">Hi,<o:p></o:p></span></p><div><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black"><o:p>&nbsp;</o:p></span></p></div><div><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black">Spark
is a compute engine. &nbsp;Ignite also provide compute feature. Also Ignite can integrate
with spark.<o:p></o:p></span></p></div><div><p class="MsoNormal"><span
style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black">We
are using ignite compute map-reduce feature now. &nbsp;It is very fast.<o:p></o:p></span></p></div><div><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black">I
am just curious how spark compares with ignite on computing.<o:p></o:p></span></p></div><div><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black">it
is possible using spark API computing ignite cache data?&nbsp;<o:p></o:p></span></p></div><div><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black"><o:p>&nbsp;</o:p></span></p></div><div
id="ntes-pcmail-signature"><div><div><div id="ntes-pcmail-signature-default"><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black">Thanks<o:p></o:p></span></p></div><div
id="ntes-pcmail-signature-default"><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft
YaHei&quot;,sans-serif;color:black">Shawn<o:p></o:p></span></p></div></div></div></div><p
class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Microsoft YaHei&quot;,sans-serif;color:black"><br><br><o:p></o:p></span></p><p
class="MsoNormal"><o:p>&nbsp;</o:p></p></div></blockquote><!--😀-->
</div>
</body>
</html>
Mime
View raw message