ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Aleksandrov <aealexsand...@gmail.com>
Subject Re: Ignite Spark Example Question
Date Fri, 09 Aug 2019 10:08:44 GMT
Hi,

Spark contains several *SaveModes *that will be applied if the table 
that you are going to use exists:

* *Overwrite *- with this option you *will try to re-create* existed 
table or create new and load data there using IgniteDataStreamer 
implementation
* *Append *- with this option you *will not try to re-create* existed 
table or create new table and just load the data to existed table

* *ErrorIfExists *- with this option you will get the exception if the 
table that you are going to use exists

* *Ignore *- with this option nothing will be done in case if the table 
that you are going to use exists. If table already exists, the save 
operation is expected to not save the contents of the DataFrame and to 
not change the existing data.

According to your question:

You should use the *Append *SaveMode for your spark integration in case 
if you are going to store new data to cache and save the previous stored 
data.

Note, that in case if you will store the data for the same Primary Keys 
then with data will be overwritten in Ignite table. For example:

1)Add person {id=1, name=Vlad, age=19} where id is the primary key
2)Add person {id=1, name=Nikita, age=26} where id is the primary key

In Ignite you will see only {id=1, name=Nikita, age=26}.

Also here you can see the code sample for you and other information 
about SaveModes:

https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-saving-dataframes

BR,
Andrei

On 2019/08/08 17:33:39, sri hari kali charan Tummala <k...@gmail.com> 
wrote:
 > Hi All,>
 >
 > I am new to Apache Ignite community I am testing out ignite for 
knowledge>
 > sake in the below example the code reads a json file and writes to 
ingite>
 > in-memory table is it overwriting can I do append mode I did try spark>
 > append mode .mode(org.apache.spark.sql.SaveMode.Append)>
 > without stopping one ignite application inginte.stop which keeps the 
cache>
 > alive and tried to insert data to cache twice but I am still getting 4>
 > records I was expecting 8 records , what would be the reason ?>
 >
 > 
https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89>


 >
 > -- >
 > Thanks & Regards>
 > Sri Tummala>
 >

Mime
View raw message