ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Valentin Kulichenko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-3084) Investigate how Ignite can support Spark DataFrame
Date Tue, 03 Jan 2017 17:27:58 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795597#comment-15795597

Valentin Kulichenko commented on IGNITE-3084:

Logical plan (which is actually AST) is built by Spark based on the API calls you make. It
supports both SQL (Spark parses it by itself in this case) and chain methods like {{filter(..)}},
{{join(..)}}, etc. Logical plan is then converted to physical plan which defines how the logical
plan is actually executed. So basically we need a strategy that will generate SQL query for
Ignite based on AST provided by Spark.

In addition to this, MemSQL provides an option to execute SQL query as is when {{SQLContext.sql(..)}}
method is called (i.e. it bypasses Spark query parser/planner). Not sure this is really useful
because this implies adding another method on top of standard API, but it's fairly easy to
add, so it make sense to do the same.

> Investigate how Ignite can support Spark DataFrame
> --------------------------------------------------
>                 Key: IGNITE-3084
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3084
>             Project: Ignite
>          Issue Type: Task
>          Components: Ignite RDD
>    Affects Versions: 1.5.0.final
>            Reporter: Vladimir Ozerov
>            Assignee: Valentin Kulichenko
>              Labels: bigdata
>             Fix For: 2.0
> We see increasing demand on nice DataFrame support for our Spark integration. Need to
investigate how could we do that.
> Looks like we can investigate how MemSQL do that and take it as a starting point.

This message was sent by Atlassian JIRA

View raw message