ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ray <ray...@cisco.com>
Subject Spark dataframe API will get wrong schema if user executes add/drop column DDL
Date Mon, 19 Nov 2018 09:53:16 GMT
Currently, when user performs add/remove column DDL, the QueryEntity will not
change.

This result in Spark getting wrong schema because Spark relies on
QueryEntity to construct data frame schema.

In Vladimir Ozerov's reply in dev list,
http://apache-ignite-developers.2346864.n4.nabble.com/Schema-in-CacheConfig-is-not-updated-after-DDL-commands-Add-drop-column-Create-drop-index-td38002.html.

This behavior is by design, so I decide to fix this issue from the Spark
side.

 

So I propose this solution, instead of getting schema by QueryEntity I want
to get schema by a SQL select command.

Nikolay Izhikov, what do you think about this solution?

I already created a ticket in JIRA,
https://issues.apache.org/jira/browse/IGNITE-10314

If you think this solution OK then I'll start implementing.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Mime
View raw message