ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ray (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-10314) Spark dataframe will get wrong schema if user executes add/drop column DDL
Date Thu, 22 Nov 2018 05:32:00 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ray updated IGNITE-10314:
-------------------------
    Description: 
When user performs add/remove column in DDL,  Spark will get the old/wrong schema.

 

Analyse 

Currently Spark data frame API relies on QueryEntity to construct schema, but QueryEntity
in QuerySchema is a local copy of the original QueryEntity, so the original QueryEntity is
not updated when modification happens.

 

Solution

Get the latest schema using JDBC thin driver's column metadata call, then update fields in
QueryEntity.

  was:
When user performs add/remove column in DDL,  Spark will get the old/wrong schema.

 

Analyse 

Currently Spark data frame API relies on QueryEntity to construct schema, but QueryEntity
in QuerySchema is a local copy of the original QueryEntity, so the original QueryEntity is
not updated when modification happens.

 

Solution

Get the schema using sql, get rid of QueryEntity.


> Spark dataframe will get wrong schema if user executes add/drop column DDL
> --------------------------------------------------------------------------
>
>                 Key: IGNITE-10314
>                 URL: https://issues.apache.org/jira/browse/IGNITE-10314
>             Project: Ignite
>          Issue Type: Bug
>          Components: spark
>    Affects Versions: 2.3, 2.4, 2.5, 2.6, 2.7
>            Reporter: Ray
>            Assignee: Ray
>            Priority: Critical
>             Fix For: 2.8
>
>
> When user performs add/remove column in DDL,  Spark will get the old/wrong schema.
>  
> Analyse 
> Currently Spark data frame API relies on QueryEntity to construct schema, but QueryEntity
in QuerySchema is a local copy of the original QueryEntity, so the original QueryEntity is
not updated when modification happens.
>  
> Solution
> Get the latest schema using JDBC thin driver's column metadata call, then update fields
in QueryEntity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message