ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikolay Izhikov (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (IGNITE-7337) Spark Data Frames: support saving a data frame in Ignite
Date Wed, 31 Jan 2018 04:34:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346248#comment-16346248
] 

Nikolay Izhikov edited comment on IGNITE-7337 at 1/31/18 4:33 AM:
------------------------------------------------------------------

Hello, [~vkulichenko]

> can you please explain all new configuration parameters you added? What are they needed
for?

* {{OPTION_CACHE_FOR_DDL}} - Ignite can't execute {{DROP TABLE}} query using SQL table cache.
So we need some other cache to make a {{DROP TABLE}} query. Also we need some cache to make
{{CREATE TABLE}} query. With this option use pass cache name to use. 

* {{OPTION_PRIMARY_KEY_FIELDS}} - Ignite can't create table without primary key. With this
option user can pass fields list to use as a primary key. For now these option is required
if data are stored in a new table or using {{SaveMode.Overwrite}} mode.

* {{OPTION_CREATE_TABLE_OPTIONS}} - Ignite accepts some extension parameters when creating
table. They are passed using {{WITH "param1=value1"}} clause. This option may contains {{param1=value1}}
string if user want to specify that extension parameters. This option can be empty. 

> But what if table already exists? 

According to Spark API [SaveMode|https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/java/org/apache/spark/sql/SaveMode.java#L27]

* {{SaveMode.Overwrite}} - existing table will be dropped.
* {{SaveMode.Append}} - data will be appended to existing table.
* {{SaveMode.Ignore}} - nothing will happen.
* {{SaveMode.ErrorIfExists}} - exception will be throwed.


was (Author: nizhikov):
Hello, [~vkulichenko]

> can you please explain all new configuration parameters you added? What are they needed
for?

* {{OPTION_CACHE_FOR_DDL}} - Ignite can't execute {{DROP TABLE}} query using SQL table cache.
So we need some other cache to make a {{DROP TABLE}} query. With this option use pass cache
name. For now these option is required if data are stored in a new table or using {{SaveMode.Overwrite}}
mode.

* {{OPTION_PRIMARY_KEY_FIELDS}} - Ignite can't create table without primary key. With this
option user can pass fields list to use as a primary key. For now these option is required
if data are stored in a new table or using {{SaveMode.Overwrite}} mode.

* {{OPTION_CREATE_TABLE_OPTIONS}} - Ignite accepts some extension parameters when creating
table. They are passed using {{WITH "param1=value1"}} clause. This option may contains {{param1=value1}}
string if user want to specify that extension parameters. This option can be empty. 

> But what if table already exists? 

According to Spark API [SaveMode|https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/java/org/apache/spark/sql/SaveMode.java#L27]

* {{SaveMode.Overwrite}} - existing table will be dropped.
* {{SaveMode.Append}} - data will be appended to existing table.
* {{SaveMode.Ignore}} - nothing will happen.
* {{SaveMode.ErrorIfExists}} - exception will be throwed.

> Spark Data Frames: support saving a data frame in Ignite
> --------------------------------------------------------
>
>                 Key: IGNITE-7337
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7337
>             Project: Ignite
>          Issue Type: New Feature
>          Components: spark
>    Affects Versions: 2.3
>            Reporter: Valentin Kulichenko
>            Assignee: Nikolay Izhikov
>            Priority: Major
>             Fix For: 2.5
>
>
> Once Ignite data source for Spark is implemented, we need to add an ability to store
a data frame in Ignite. Most likely if should be enough to provide implementation for the
following traits:
> * {{InsertableRelation}}
> * {{CreatableRelationProvider}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message